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ABSThACr 

A pilot^mole^ for evaluating state-of-the-art 
short-courses for engineer practitioner.:, was developed based on 
asoessments of 10 Continuing Education in Engineering (GEE) courses 
offered by University Exteusion of the University of California, 
BerKe^ey« Instruments used m the study included student 
questionnaires, faculty interviews, follow-up questionnaires and 
interviews with participants, interviews with supervisers and 
colleagues ana ooservations of classes. Program characteristics, 
demographic and attitudinal participant characteristics, and work 
deniand and conditions were examined. ParfOne, Procedures, examines 
course characteristics, design of study, questionnaires, evaluation 
procedures, and interviews. Part Two, Results, examines developing a 
data base, students* rating by course, student profiles, experience 
vs. attitude toward the course, course expectations, and student 
characteristics. Ihe final sections offer a summary of findings such 
as: student diversity poses a major dilemma for faculty members who 
plan to teach CEE short-cc urses; participants were generally 
satisfied witn tae CEE courses tLey took; and participants were 
iLterested in taking furtner courses in the subject. A discussion 
section ofr.ers general information on aspects of planning, teaching, 
and evaluating a CEE shor fcouri^e and provides a taxonomy of 
educational outcomes. Rererences are provided. Appendices include: 
instruments used m the study; example of a data base: and evaluation 
rt^^ ^rts on individudi CEE courses. (LC) 
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ABSTRACT 

Maxwell, Martha. An Evaluation Model for State of the Art Programs for 
Engineer Practit-ioners , (NSF Grant # SED -78-22138), University Extension, 
University of California at Berkeley, March 1980. 

An pilot-model for evaluating state of the art short-covrses for 
engineer practitioners was developed based on assessments of IC Continu- 
ing Education in Engineering courses offered by University Extension 
at the University of California, Berkeley. The instruments used included 
student questionnaires, faculty interviews, follow-up questionnaires 
and interviews with participants, interviews with supervisers and collea- 
gues and observations of classes. A sample data base, a taxonomy of 
educational outcomes, and suggestions for improving CEE courses are 
described. 
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APPENDIX C Evaluation Reports on Individual CEE Courses 



The purpose of tixis project was to develop a "pile t-model" for 
evaluating the effectiveness of state-of-the-art short courses in 
Continuing Education for Engineers (CEE) . Specifically, we attempted to 
identify and document evidenc^ of participants' learning and transfer 
of new information, concepts, and technology taught in CEJ programs and 
identify the variables that appear to enhance or impede application of 
the skills and information learned. Among the variable studies were: 
(1) program characteristics (including organization, content, teaching 
strategies, etc.); (2) demographic and dttitudinal participant character- 
istics (experience in course .specialty, academic preparation, expecta- 
tions, attitudes toward course, etc.) and (3) work demand and conditions 
(data on work context from participants, colleagues and supervisor.) 

State of the art CEE short-courses are operationally defined in this 
study as briof , intensive programs that focus on highly specialized topics, 
that are usally taught 6y instructional teams (including both engineering 
faculty and representatives from industry,) and that emphasize new 
developments in theory, resea^cn, and technology. State-of-th« art shor- 
courses are distinguished from review or refresher courses designed to 
help engineers prepare for Board examinations or those offered to students 
seeking colloge degrees. Although there is some content overlap between 
state-of-the-art short-courses and other CEE courses, we view state-of- 
the-art courses as being more cleeorly focused on a relatively narrow, but 
highly significant topic. They are not seminars among peers as are many 
programs offered by professional associations, but are- rather coui'ses in 
which instructors trartslate new information in "bit-sized" pieces for engij'neer 
practitioners who are not as sophisticated about the topic as are 1 .turers'. 

(1) 
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Because of their nature, state of t;he art courses may have restricted- 
audiences, small enrollments, and or may attract engineers from distant 
geographical areas. 

i' 

The project produced a pilot-model for evaluating the effectiveness 
of CEE State-of-the-art short-coursa^ and set of suggestions for program 

directors and instructors to use as a guide for planning and improving 
CEE courses. 

PROCEDURES 

To implement the project goal, an intensive review of the literature 
and a study of CEE short-courses and their participants were cpinp.leted. 
Twelve sections of ten different short courses offered by the University 
of California, Berkeley Extension Department and the College of Engineering 
in 1978 and 1979 were evaluated using the following procedures: 
1) faculty members were interviewed prior the beginning of the course 
or contacted by mail; 2) a student evaluation questionnaire was 
tailored the the special needs of the course and approved by the faculty 
member in charge; 3)" student evaluation questionnaires were administered 
to students during the last day of the course and/or sent to them after 
the course ended; 4) observations of the class sessions were made by 
the evaluation team; and 5) case studies based on follow-up interviews 
with participants who volunteered, and, in some cases, their colleagues 
and administrators were produced. 

The ten short courses thosen for the study are typical of the wide 
range of courses offered by the University of California, Berkely Exten- 
sion's Engineering Department during the period covered by the grant. 
These courses differed in many ways— in length (from one to five days). 
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TABLE 1 
COURSE CHARACTERISTICS^ 



Course Title & Ijescription 



Prerequisites 



Teaching 
Methods 



Length 



No, of 
Speakers 



Materials 



Interfacing to a Microcomputer; 
Overview to input/output int>er- 
facing techniques & hardwate/ 
software tradeoffs in using • 
microcomputers , 

CoCT^mter Aids for IC Tech- 
nology and Device Design ; 
Current problems in IC pro- 
cessing^ update on Stanford 
research^ & advances in pro- 
cess modeling. 



Background in 
engineering or 
science . Some 
knowledge of 
progr aiming 

None Staced 



Lectures plus 5 days 

workshop-daily . 
lab. Practice on 
6 experiments. 



Lectures, a/v aids 1 day 
including videotape. 



2 and / yes 
2 lab a$sts. manuals 



13 Yes- background 

material/ (Lecture 
* notes as adver- 
tised were not 
avail£|ble«) 



Exponential Smoothing and 
Adaptive Forecasting Tech- . 
niques ; Designing & applying 
simple & useful forecasting 
procedures for engrs, & busi- 
nessmen. 

Value Engineered Design & 
Construction ;- How to accon\p- 
lish design & construction 
saving usxtl^ the value engi- 
neering management system. 

Composite Materials Computation 
Workshop ; A Practical Guide to 
Design and Testing; The most 
current guide to solving prob- 
lems in composites; design & test- 
ing for users & producers. 



1V> 



Some familiar- 
ity with basic 
statistical con- 
cepts 



None stated 



Operational know- 
ledge of TI-569 



Lectures, question 
& answer session 



1 day 



yes 



Leccures, films, 
t^ifiun workshops 
discussing real- 
life projects. 



Lectures, drills, 
2 informal 2-hr, 
workshop sessions 
each evening. 



5 days 



days 



Ves-books & 
raaterials sent 
in advance of 
course. 



Yes-students 
given methods 
charts, & pre- 
programmed equa- 
tions for calcu- 
lators' £> handbooks. 
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TABLE 1 
COURSE CHARACTERISTICS 
(Continued) 



Courae Title & Description 



Prerequisites 



Teaching 
Methods 



Length 



No. of 
Speakers 



Materials 



Fundamentals of High-Resolu- 
t ion Lithography: A concep- 
tual foundation for 3 types 
of Lithography- Deep UV, B-Beami 
& x-ray, resist materials, 
compatible processes, & process 
modeling tools, & a user- 
oriented simulator for-project 
or printing. 



None stated 



Lectures, ques- 
tion & answer. 
Panel at end, 
a/v aids. 



1 day 



Yes, (Notes 
& some view- 
graphs. 



Klectron-Beam Lithography; 

Advantages & disadvantages of 
electron-beam tabrlcation for 
high performance integrated 
filectronic circuits. 

Airport Systems Planning & 
Desiqy^: Air travel demand 
forcasting, site selection- 
capacity; airfield design 
passenger processing, finan- 
cial planning, & current 
legislation. 



None stated 



Ncne stated 



Lectures, ques- 
tion & answer. 
Panel at end, 
a/v aids. 



Lectures, a/v 
aj ds, discussion 
session. 



1 d^y 



4 days 
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Yes 



Yes 
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Kngineerinq Design in Ti.'iiber ; 
The technology of designing struc- 
tures in timber. 

KartlKiuake Analysis of Multi- 
stoty Fram fei Shearwall mild- 
intj s : Describes 4 computer codes 
for earthquake analyses & their 
practical applications. 



1^ 



General knowledge of 
structural theory 
& design. 



Lectures, a/v 
aids. 



3^ days 



Some experience with Lectures, ques- 

computer applications tion & answer session 2 days 

for structural design 

& fundamental building 

analysis. 



Handouts 



Yes, input 
manuals, ex- 
planations or 
each code & 
examples. 
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in subject area (from high resolution lithography to airport systems in 

instructional format (lectures to intensive workshops) , and in size 

(from 12 to more than 300 students). (See Table 1 Course Characteristics.) 

DESIGN 

IS. • 

Fig. 1* outlines the general evaluation plan and lists the major 
coui'se aspects addressed in the left column: program objectives, 
instructional methodology, program context, student outcomes, overall 
effectiveness, side effects and cost factors. The instruments are given 
at the top of the chart with an X indicating which methods were used 
to measure each concern. The instruments included: 

^- Systematic expert judgment- decisions made by the faculty sponsor; 
the CEE coordinator and the instructors in designing and planning the 
program. 

2* Student evaj.uation questionnaire- administered to all participants 
at the end of each program. 

3- Instructor evaluation interviews- conducted with instructors prior 
to and, in some cases, after the course. 

4. Observations of course- the investigators observed some of the sessions 
of each program when possible. 

5. Follow-up Survey Questionnaires- mailed to students after the course 
was completed. 

6. Interview with Students- volunteers were contacted by phone and inter- 
views arranged after the course ended. 

7. Interviews with Administrators- arranged by phone with participant 
consent. 



*This plan is a modified version of evaluation methods described by 
Scriven (1974) and Anderson and Ball (1979). 
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FIGURE 1 
DESIGN OF STUDY 
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Figure 1 Continued 
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QUESTIONNAIRES 

Each cl dss wds suirvcycd 6lth6r dt th6 conclusion of th6 s6Sslon 
nr after the ^course had ended through follow-up questionnaires* 
Whether end-of-course or follow-up surveys were used was determined by 
the instructor for in one-day courses there was rarely enough time to 
administer questionnaires 

The student evaluation questionnaires were designed to measure the 
following variables : 1) demographic characteristics of the student/ 
participants, 2) satisfaction with attitudes toward various aspects of 
the course (e.q. comparision with other CEE coursf^s, whether the course 
fulfilled expectations, etc.,) 3) suggestions for improving the ^urse, 

4) relevance of course materials and content to engineers' needs, and 

5) impact of the course and how information and methods learned in the 
course were or could be used on the job. (NOTE; questions may be 
found in Appendix A) Complete questionnaires are in Appendix. C. 

Questions were tailored to the individual course and, therefore, 
varied somewhat from course to course. Each course questionnaire 
was reviewed by the faculty coordinator and revised as necessary before 
administering it to the participants. 

Questionnaires were given at the end of the course to six classes 
and stuuent in four classes were sent follow-up questionnedres at periods 
ranging from two weeks to one year after th,*^ course was completed, 
(see TaUDle 2 for the evaluation procedures used in different classes.) 

Responses to the questionnaires werr^ coded, punched on IBM cards, 
and analyzed on the IBM 6400 Computer using the Statx^ Jical Package for 
the Social Sciences (SPSS) developed at the Vogelback Computing Center at 
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TABLE 2 
EVALUATION PRCX:EDURES 



CX)UkSE 



NUMBER 
ENROLLED 



INSTRUCTOR 
INTERVIEWED 



SURVEY OP STUDENTS 
END OF CLASS FOLLOW-UP 
(») (#) 



OBSERVATION OF 
OF CLASS 



STUDENTS 
INTERVIEWED 

(0) 



Interfacing to a 
Microccmputer 

Computer Aids for 
IC Technplpqy 



12 



305 



Forecasting Tech- 
niques 36 

Value Engineering 

(1978 Class) 11 
(19 79 Class) 17 

Compr^site Materials 
Computation Wor)cshop 65 

Fundamentals of High 
Resolution Litho- 
graphy 132 

E-Beam Lithography 108 

Airport Systems 
(1978 Class) 48 
(1979 Class) 46 

Engr, Dos\ in Timi^r 28 

Earthquake A ^/Aysis 50 



yes 



yes 



yes 



yes 
yes 



no 



yes (12) 



no 



no 



no 

yes (17) 



no 

yes (90) 

19) 



no 



yes (52) 



yes (19) 



yes (8) 

no 



yes (35) 



no 

yes (54) 

yes (27) 
no 

no 

no 



yes 



yes 



yes 



no 
yes 



no 

yes 
no 

no 
yes 

yes 

yes 



yes (4) 

' 1 

yes (5) 



yes (2; 



yes (3) 
yes (4) 



yes (7) 



yes (20) ^ 
1 

yes (7) 



no 

yes (9) 

yes (9) 

yes (11) 



Number of stu;!«nts replying to survey qu« 
Ni^mber of follLW-ui> interviews with ^tuden: . 



1. Administrator5:/''.olleagues J.nterviewed (tote 



1 



Questionnaires .- ..ia seit to a random sample xu^ students; 52 replied 
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Northwestern University. 

Participants volunteered to be interviewed for the project by inclu- 
ding their names and phone numbers on the end of the quest ionnare. 
(See Appendix A for the statment describing the project and the interview 
invitation. ) 
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INTERVIEWS J 



Participants who were willing to be interviewed or wanted more 
information about' the study were called by a project team member, and 
interviews were arranged. In most cases, participants requested phone 



\ 



interviews. 



The qu^^tions asked by the interviewer were designed to encourage 
students to elaWrate on and clarify their questionnaire responses, to 
describe whether and how they had applied course information to their 
work, and to describe the context of their work environment and its 
impact on transfer. (The Interview Protocol is included in Appendix A.) 
Interviewers also asked p€u:ticipants for permission to interview their 
supervisors and colleagues. 

Faculty Interviews In most courses it was possible for a member of the 
project to interview the faculty members before the course began. In 
other instances, the faculty member was contacted during or after the 
course, and queried about the course goals, the instructor's expectations, 
etc. (The faculty interview protocol is included in Appendix A.) 
Interviews with Supervisers and Colleagues , if the participant agreed, 
the project team member contacted the superviser and/or colleagues mentioned 
by the interviewee. The protocol for these interviews is found in Appendix 
A. 

Analysis of Interview Data. Interview responses of participants, colleagues, 
and supervisers were recorded, summarized for the individual course eval- 
uation reports (See Appendix C) , coded for the con^juter analysis, and used 
to write brief case stuJies. 

Faculty interview data were , recorded and summarized in the individual 
evaluations and used in inte^reting the results of this project. 



course 
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Class Observations 

A project team member attended the CEE classes i recorded impressions, 
and talked with students when this was feasible (See Table 2 }. These 
observations were incorporated in the individual course evaluations found 
in Appendix C and aided us to refining the questionnaire^ and irterview 
protocols and in interpreting results. 

RESULTS 

Of the 653 students' from 12 classes who received course evaluation 
questionnaires, 423 replied. As might be e)q)ected, the response rate was 
higher when questionnaires were given out during th3 class (80%) than when 
they were sent by mail (52%) • We tried to increase the return rate of 
mailed questionnaires by following the suggestions of Ohrlich in Designing 
Sensible Surveys , 1978 (p. 94) -that is, a follow-up postcard was sent 
within one week; a second copy of the survey and cover letter was sent 
within two weeks; and a third instrument was sent within one month. However, 
when this intensive follow-up was used on two classes (E-Beam Lithography 
and Workshop on Composite Materials), the response rate did not signifi- 
cently excede that of courses where participants were not contacted 
repeatedly, and the small number of additional responses obtained did not 
seem to warrant the high mailing costs. Therefore, we decided to send only 
one follow-up - a cover letter and second copy of the questionnaire. 

Although the response rate was higher for questionnaire^ given out 
in class, these participants tended to reply to the questions more 
superficially and leave more items blank than did students who returned 
follow-up questionnaires by mail . 119 respondents indicated they were 
willing to be interviewed and 81 were interviewed by a member of the project 
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team. A number of factors mitigated against interviewing all of those* 
who were willing to participe.te - sometimes it was not possible to 
contact the students. Sometimes they changed their minds about being 
interviewed. Others were too busy- or had changed fields, and in some 
cases/ we felt that we had interviewed enough students from a course 
to sample the range of opinions in the class. 

From computer analysis of questionnaire cUita we prepared evalua- 
tion reports on each course. (See Appendix C) . We compared courses, 
and developed a model data base useful for assessing new courses and making 
other kinds of administrative and instructional decisions (See Appendix B.) 
In addition, we analyzed the profiles of students with different 
characteristics. % 



14 




Developing a Data Base.' A Sample data base was compiled by.. summing the 



responses of participants to the written survey and interview questions 
and computing percentages. (See Appendix A.) Such a data base could be 
of great value in aiding administrators in making decisions and in help- 
ing new instructors form realistic expectations about CEE courses, if 
kept systematically over a longer time-span. A systematic data base 
provides a standard for identifying succes^iul courses and analyzing their 
characteristics r and for assessing the effectiveness of new courses and 
the improvements made in old one?. Courses that are not working well. can 
be identified and the reasons explored. Also, the.-:* data reveal differ- 
ences in the effectiveness of different courses or repeated offerings 
of the saxae course. 

New instructors might find data base information valuable in planning 
courses^ for example the fact that one-third of the engineers in CEE classes 
have taken no previous CEE courses, or that on the average about a quarter 
of the students have had no experience in the specialty addressed by the 
course or that only 60% find ideas that they can use immediately and ^ 
the majority are mainly interested in one third of the material presented. 
It might also be helpful to know that about 6 percent of the students 
rate the course 'worse than other CEE courses they have taken. 

Ideally, a cumulative data base would be developed for each course 
because the participants in different CEE courses vary greatly in their 
characteristics and responses. To illustrate how this might work, consider 
the finding shown in Taole 3 which shows how participants in different 
courses responde^l to the question, "How did this course compatffe with other 
CEE courses you have taken?" 
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Table 3 



Students'. Ratings by Course 



course' 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


TOTAL 


Percent of students 
rating it in compari- 
son with other CEE 
courses taken 
























Best I have taken 


8% 


NA 


0 




14% 


5% 


4% 


5% 


18% 


4% 




Better than others 


50 


NA 


18 


44 


30 


24 


30-- 


40 


4 


25 


30 


About the same 
as others 


0 


NA 


17 


28 


30 


12 • 


31 


•24 


35 


38 


25 


Worse than others 


0 


NA 


18 


4 


0 


2 


9 


8 


25 • 


13 


6' 


Have taken no others 


42 


NA 


47 


12 


26 


57 • 


26/ 


23 


18 


20 


33 





Code for course^: 

1« Interfacing to a Microcomputer 
2* Computer Aids for IC Tech. 
3» Forecatsting Techniques 
4« Value Engineering 
5» composite Materials 

6« Fundamentals of High Resolution Litho. 
■ 7* E-Beam Litho* 

8« Airport Systems & Design 

9« Timber Design 
lO Earthquake Analysis 



. More participants gave higher ratings to the courses in microcomputing/ 
conposite materials, valXie engineering, amd airport systems, than to other courses* 
Each of the preferred courses lasted a week and with the exception of the air- 
port course, involved hands-^n lab. work or participation on teams to solve 
real-life problem. Also, these courses Were smaller in Jize (less than 50 
students) than some of the others. Students in Timber Design and Forecasting 
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Techniques rated these courses lower than did participamts in other courses, 

signaling that there were problems. The Forecasting course was a one-day 

course attended ty a very wide range of students — i.e., from beginners 

to sophisticated conputer science graduate students — and the professor had 

a most difficult time teaching to such a diverse group. (See report in 

Appendix C.) In this cace, rai^log the stated prerequistcs from "a general 

understanding of basic statistics" to something higher and altering the 

instructional strategies used would improve the course for both the professor 

and the students. 

The Timber Course, which was of f ered for the first time, attracted 

students Who had taken previous CEE courses l/ut were new to this rspecialty, 

\ 

(40%) and engineers conplained that the course was' too theoretical. Handouts 
but not course notes were given out and' a number of students objected to not 
receiving notes.' 

Differences between courses in the percentage of students who havi not 
taken previous CEE courses dre also apparent in Table 3. Interfacing to a 
Microcomputer, Foreca^^^g Techniques, arid Fundamentals of High Resolution 
Lithography attracted higher percentages of students new to' CEE, than did the 
other courses. Perhaps this resulted from the newness of the subject matter 

cover^.d and in the case of High Resolution Lithography the course title 

\ 

"Fundamehtals" attracted a number of students who wanted a beginning course. 
(See discussion in App. C.) ^ " * 

Table 4 illustrates another way to use questionnaire data from individual 
courses to show the relationship between variables. 
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TABLE 4 

Relation Between the % of Engineers with 
Minimal Experience in the Specialty and 
Coitplaints about Instxuctora' Emphasis on Applications 



Course 



% of participants 50% NA 
with less than 1 
yr's. experience 
in specialty 

% of participants NA NA 
stating that 
instructors 
placed too little 
emphasis on appli- 
cations 



10 



10% 12% 42% 27% 29% 29% 40% 



26% 



57 14 61 NA 27 29 53 
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Code for courses : 

1 Interfacing to a Microcomputer 

2 Computer Aids for IC Tech 

3 « Forecasting Techniques 

4 a Value Engineering 

5 « Composite Material 

6 = Fundamentals of. High Resolution Litho 

7 » ErBeam Litho. 

8 « Airport Systems 

9 Timber Design 

XQ = Earthquake Analysis 



The percentages in Table 4 suggest a relationship between the number of in- 
experienced participants enrolled in a class and con¥>laints that the instructors 
did -not placL enough emphasis on applications* In both the Composites Materials 
and Timber courses there were a high percentage of newcomers to the specialty, 
and over half the class felt there was too little emphasis ^on applications. 
(See Appendix A and the Discussion Section) In the Forecasting course, altliough 
most of the students had practical experience in the field, only a few had formal 
courses and it was to these advanced students that the professor directed his 
presentation. As a result both advanced and beginning students were turned off. 
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(See Appendix C.) m the Microcomputer course most of the students, des- 
pite having little experience with this parUcular device, did have ex- 
perience with computer programming in general. Although the question 
about instructors' emphasis on applications was not asked and most stu- 
dents were favorable toward the course, a number complained that there 
was too much emphasis on hardware. 

Evaluating CEE courses as different in content and emphasis as these 
courses were which attract suCh diverse student is difficult indeed. The 
many variable involved (student characteristics, type of course, length 
and size of class, instructors' style and level, etc.) all interact. 
However, a data base can provide the program planner with a barometer to 
measure the relative/succ6ss of different classes. Both the strongest and 
weakest courses can be identified and their characteristics analyzed and 
studied, so that models of succiisful courses can be developed and ame- 
liorative steps can be taken to strengthen weak programs. 
Student Profiles. In addition to showing us overall characteristic s of 
CEE students and comparing courses, the SPSS data analysis program enabled 
us to develop profiles of students. These profiles yield a partial answer 
to questions about the characteristics of CEE s^ndents. For example, we 
were interested in the fact that a number of engineers with little or no 
experience in the topic enrolled in very specialized CEE courses and 
wondered why? Were they merely curie us? Were they contemplating changing 
careers? Were they new graduates just embarking on an engineering career? 
To find out, we analyzed profiles of the responses of students with vary.i ng 
levels of experience in the course topics. 
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EXPERIENCE VS ATTITUDE TOWARD COURSE (PROFILE) 



One suprising fact that emerged from the data analysis was that 72 
percent of the participants who had little or no experience in the course 
specialty had completed their degrees more than 14 years ago. 92% of those 
new to the field had held engineering degrees for more than 4 years. This 
suggests that some had made mid-career changes or were contemplating doing 
so or had some other reason to be interested in the course. For example, 
one was a company president* who was trying to keep abeast of new develop- 
ments in one of his departments, another a structural engineer curious 
about whether computer programs in earthquake analysis might have implications 
for his work in designing r.tacJcs to withstand wind factors. 



Another /^as a consultant, trained in a different specialty, but convinced 
that he could save money by taking the course himself rather than hiring 
a consultant and improve his advice to his clients. 

The data analysis revealed some significant differences between 
participants with little experience in the course content and. those with 
more experience. The inexperienced group were more likely to hold Ph.D. 
degrees, spend less than 30% of their time on ddminist-.rative duties and more 
than 70% of their working time in technical tasks, primarily in research 
and development units. The average inexperienced participant rated the 
course as meeting his goals in updating information, learning fundamentals, 
acquiring a perspective for decision-making as "somewhat successful," 
but felt it improved his general background "very well". 84% had no 
previous courses in the field conpared with 46 . of the more experienced 
engineers. Furthermore, the inexperienced. group showed less interest 
in taking other CEE courses, And were not planning to do research on their 
own. 
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They conplained more frequently about course materials as well as most 
other aspects of the course. They felt instructors did not place enough 
^nphasis on application , nor understand engineers needs. 75% of the 
group with little experience felt unprepared for the course compared with 
33% of the more experienced engineers. A larger percentage of the in- 
experienced group (60%) said they found nothing in the course that they could 
use immediately compared with 40% of the e5q>erienced group. Inexperienced 
participants were more likely to feel they might be able to use the course 
information in the future. 

In summary, the Inexperienced groip tended to be highly educated 
but unprepared for thi$ course, complained aOx^ut the instructor ' s . not 
emphasizing applications, the materials, amd other aspects of the course, 
and found little they could use immediately. They were also less l:.kely 
to volunteer for interviews than the more experienced group. 

Anothoi qaescior we examined was expectations for the course since 
this question is oftcA u.^od to assess attitudes toward a course. 
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COURSE EXPECTATIONS 

The profiles of engineers who said that the course did not meet 
their expectations differed from those whose ejqjectations were met. The 
former group were: 

— mpre likely to hold a Bachelor's degree than a Ph.D., to work primarily 
on technical tasks, and to have had more than 10 year's e3?>erience in the 
field. 

— more likely to have enrolled in the Forecasting Airport Systems and 
I.e. Technology courses than the other courses. 

— more likely to have taken previous GEE courses and more likely to rate 
this course as "worse," "better," or "best" in comparison with other 
courses they had taken. (Fewer ranked it as "the same" as other courses.) 
— less likely to find topics that they could use immediately or in the 
future and were general less satisfied that the course met their goals of 
updating, learning fundamentals, and solving prot>lems. 

— gave more suggestions for improving the course especially about instruc- 
tors and content and e^ressed more negative evaluations cf instructors. 
They tended to criticize instructors for not placing enough' emphasis on 
applications' (74%)^ not focusing on engineers' interests, not placing' 
enough or placing too much emphasis on fundamentals. 

— they were more likely to discuss the course with their colleagues but 
less likely to recomjnend this course to their peers. 

— they agreed that the course had helped them professionally in aUxDUt the 
same proportions as those whose expectations were met. 

In suirmary, the data base can be used to determine characteristics of 
engineers who enroll in CEE couraes, to determined differences between 
classes, and to make profiles of satisfied and dissatisfied students. 
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Characteristics of Students Who Were Interviewed 

Twenty-two percent of the students who con?)leted written question- 
naires were willing to be interviewed for the study and 68 percent 
( 81 engineers ) were interviewed . 

We compared the questionnaire responses of engineers who volunteered 
to be interviewed with those who did not to see whether .the two groups 
differed. We found that those who were willing to be interviewed were 
more involved an^ concerned about the course and had stronger opinions J 
(both positive and negative) than those who did not volunteer. That is, 
they more often rated the cours* as meeting their goals "very well" or 
"not at all" and less frequently checked "somewhat," Compared to the 
group that was not willing to be interviewed, more held Bachelors or 
Masters degrees and fewer had P h.D.s. Fewer had tal:en previous GEE 
courses and they made more suggestions about improving course materials. 
They also were more likely to have expected something different than the 
course offered. However, some were dissatisfied with the results of the 
course and others were pleasantly surprised. They also expressed a 
greatf^r interest in taking fuiure GEE courses than those who were not inter* 
viewed. 

Interviews with Admin ist ^ators . 

Five administrators who supervised course participants were interviewed 
about the effects of tJ'-a course on the engineer's work. In general, these 
supervisors had positive things to say about the 0,C,B. CEE courses, but 
little specific in .ormation about how the information learned had been 
applied. Their comments were included in some of the short case studies 
incorporated in the reports in Appendix C and in the Discussion Section. 
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2. Student Satisfaction . j " 

Generally, participants were satisfied with the CEE courses they • 
•ook. Seventy-five percent took the courses to update their knowledge 
of the specialty-, and 55 percent of these %#ere well satisfied. More 
than half of those who had taken previous CEE courses rated their pre- 
sent course as better than other courses they had taken. 

In general, engineers ''rated the intensive one-week hands-on lab. 
or problem solving courses higher than the large one-day lecture/demon- 
stration courses. 

Those whose goals were learning fundamentals, acquiring a perspec- 
tive for decision-making, solving special problems, etc. were somewhat 
less satisfied with the courses than those specifically interested in 
updating information, but even so, the average participant was satisfied. 

3. Interest in Taking Further CEE Courses. 

One consistent finding across courses was tiiat participants were 
interested in taking further courses in the subject. Completing a CEE 
course seems to arouse an interest in learning more about a topic and 
pursuing it more deeply. Consistent with the finding of other studies 
in continui.ng education, the practicing engineer who takes a CEE course 
develops a greater awareness of the subject and a desire to learn more. 
However, his/her options for further CEE courses are limited at U.C. Berkeley. 
There rs a great need for more intensive courses in all specialties 
offered-a need that is not being met by present courses. Taking a CEE 
course seems to whet the student's appetite "-jr more learning, but at 
present, there are few relevant courses offered. Many of the participants 
in current courses would be willing to spend an extra day if they could get 
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SUMMARY OF FINDINGS 

The constraints and limitations of the design and sample used in 
this study suggest caution be used in generalir.ing the following findings: 
1. Student Diversity 

Engineers and others who enroll in the University of 'California, 
Berkeley CEE short-courses vary widely in educational backgroundr exper- 
ience ^ needr e)?>ectationSr and their reasons for t^Ocing the course. They 
range from the merely curious to the college professor: from those who 
are just entering the specialty to researchers whose experiments are 
contributing to new knowledge. Three-fourths of the participants, in 
the typical course r have not taken a previous course in the specialty , 
although about two- thirds have tadcen other CEE short-courses. About a 
quarter of the participants in the average class are new to the field# 
but almost half of the participants in some classes have little or no 
experience in the specialty addressed by the course. Participants show 
a wide range of interests. Usually they are mainly concerned with a 
third of the topics covered in the course. Student diversity poses^ a 
major dilemna for the faculty members who plan and teach CEE short-courses . 

On the other hand, short-course participants share some things in 
common. They are generally well-educated, motivated to learn, and inter- 
ested in the course. In almost all cases their conpanies pay their 
course fees and expenses and the majority of students are self-selected. 
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an intensive workshop or a lab. that would help them learn how to apply 
the computer programs described or whatever the course covered. The 
desire for additional courses was equally strong among engineers taking 
one-week short-courses as those attending one-day programs. 

IMPACT OF THE COURSE 

The effects of the course on students were as varied as the students 
themselves. Ajx)ut 60% found something they could use immediately and 
55% found something they felt would be of future value (as if they w^ire 
tucking ideas away in their intellectual hope chests) . . , 

The follow-up revealed that some had applied the ideas and techniques 
learned in the course directly to their work while others had not thought 
about it since the course ended. Whether an engineer used course informa- 
tion depended on his work environment, specifically whether he had the power 
to change things or try new ideas and approaches an-f, whether he was pre- 
sently involved in projects that were relevant to the course content. 
Also application seemed to depend on the nature of the course. For example, 
the Value Engineering course participants in most cases felt the informa- 
tion they learned was very useful i*. thtir own day-to-' ay work regardless 
o:: whether the company accepted the VE package and iregardless of what their 
work colleagues were doir^/. 

Whether a course was considered practical also was contingent on the 
work setting. For example, one engineer considered a topic so futuristic 
as to be classed as science fiction relative to his corpany while another 
member of the same class went back to his lab., built an instrument, 
replicated one of the operiments described in the course, and preceded to 
determine whether it would improve his company's product. 
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The time between taking the course and being" interviewed made a 
difference. Figures, facts, and faces fade fast but feelings about' the' 
course linger on. Most remembered the course as a good ejqperience, though 
a f«w felt frustrated. Some who rated the course "about the sa^e as others 
I have taken" at the end seemed to mellow when they discussed it a few weeks 
later. Their first reaction to the course ^ they said, was based on the 
last speaker who did not tell them anything new. Several weeks later, 
they were more positive about it since they could see it in perspective. 
This suggests that end-of-class questionnaires may not have the reliability 
expected and other strategies should be used cs well. 

Recollection of the topics presented in the course after several months 
depended on whether the participant had taken subsequent CEE courses. If 
they had, they were likely to recall less about the particular course we 
were interested in and confuse the courses. 

INTERESTS IN OTHER COURSES 

Forty-five percent of the participants listed other courses that they 
would be interested in taking while 72% checked that they planned to take 
further courses. An even higher percentage (77%;» indicated that they 
planned to do further research on course topics on their own time. 

The average participant (61%) found something in the course that could 
be uscid immediately while more than half (55%) said they felt they learned 
information that they oould use in the future. 

Ripple Effect Almost all of the participants said they planned to 
talk about the course with their colleagues (97%) and with their super-- 
visors (96%). We did not ask how many planned to discuss course informa- 
tion with their subordinates, but perhaps we should have. 

ERIC 
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The follow-up interviews revealed that participants did spread the 
information learned in the course to others in their companies in a 
variety of ways - the most frequent was through memos, meeting, passing 
course materials around, holding brainstorming sessions, etc. It is 
clear that ideas and technology learned in the CEE courses are transmitted 
by participants to others in their companies. The number of non-students 
reached depends on how many interested specialists are in the partici- 
pant's department. 

Typically, participants indicated that they ejqjected to use the 
course materials regularly (54%), though 37% said "maybe" and 10% doubted 
it. Responses to this question, as is true of other questions, varied 
considerably from courses to course. 

On the whole, participants felt strongly that taking the CEE course 
would help them professionally (79%), however they were not as sure about 
the value of their taking the course to their companies. Sixty-four 
percent felt it "maybe" of value to their company, with only 29% respond- 
ing with a definite yes. Again those engineers enrolled in courses like 
Value Engineering were more positive. For example, some reported that 
if they became certified as V.E.'s their conpany could qualify for govern- 
ment grants. 

Does taking a CEE course im prove an engineer's chances for promoMnn? 

We asked five classes "How does your conpany recognize attendance 
at courses like this one?" Very few (between zero and 8%) of the partici- 
pants checked the alternative "Course attendance increases the possibility 
of promotion/raises," It appears that either few engineers view atten- 
dance at CEE courses as a way of moving up in their companies or they 
are modest about their intentions in taking the course. In a large 
follow-back study of CEE students, Morris (1978) concluded that engineers 
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who enrolled regulary in CEE courses increased their earnings over 
those who did not. We did not address this question directly as our' 
design did not include a control group. One factor that could inJ^ 
ence earnings is that CEE participants are usually self-selected, and 
it may be that better qualified people seek out courses and promotion 
follows . 
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SOURCES OF DISCONTENT 
Course Title and Descriptions 

More dollars, time, effort, and creativity seem to be invested in 
the artwork and layout of the course brochure than are spent in devising 
the course title and writing the course description, CEE course titles, 
like their academic counterparts, once approved, seem to be engraved in 
stone. They are rarely changed despite changes in instructors, content, 
and emphasis* Engineers tend to interpret these literally* First-timers 
in CEE courses complained that in some cases the course titles and des- 
criptions were misleading. For example, students who took Fundamentals 
of High Resolution Lithography for an overview of current developments in 
the field tended to be well satisfied while those who enrolled to get the 
the basic fundamentals of a field new to them were less satisfied with 
the course* Of course, the term "Fundamentals" is relatj^ve — what is con- 
sidered fundamental to a Berkeley engineering professor might be viewed 
as very advanced to a practicing engineer* 

In courses with double titlea such as Computer Aids for ic Technology 
and Device Design or Airport Planning^ and Design, engineers expected 
equal time and emphasis would be placed on both topics. In these instances, 
the device designers and the airport designers felt their areas had been 
slighted in the instructors presentations* It may be necessary for 
Extension programs to be broad in scope and cover more than one specialty 
in order to insure that courses fill or it may be that the faculty members 
feel that closely related specialties should be covered in one course, 
but, if this is true, particip2mts did not seem to understand the reasons* 

Course descriptions also came under fire by some participants* 
One intensive five-day workshop advertised as "ejq)licity application oriented 
drew coanplaints from 56% of the class 
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that the instructors did not place enough emphasis on applications. 
Clearly here there was a difference between the instructors' percep- 
tions of "application" and the students' perceptions. In other courses, 
students complained that discussion groups described in the course 
announcement were not held or that the actual course differed in signifi- 
cant ways from the brochure statements. Very few of the C2E courses we 
evaluated listed prerequistes or described the level of the course pre- 
sentations, and in some cases, the instructors overestimated the sophis- 
tication and knowledge of their classes. Although CEE courses can't 
deny access tc students, a clear description of the background knowledge 
expected by the instructor whould help the students put the course in 
proper peraipectivn.. These examples suggest that course titles and des- 
criptions should be more closely monitored by CEE director who could pro- 
vide faculty with information on how to awid these difficulties, and 
write more accurate titles and descriptions. 
COURSE MATERIALS 

Engineers who take CEE courses expect to receive clear, complete 
lecture outlines and other relevant materials * Indeed a ^;tumber of partici- 
pants enroll in CEE courses primarily for the materials. Complaints about 
inadequate or r^J^ly prepa::ed materials are the most frequent criticism 
raised by students. Ph.D. engineers, particularly those who are new to 
the specialty, are most likely to complain about the course materials, 
especially if they do not received good lecture outlines. They find it 
impossible to cake adequate notes in the information-intensive, fast- 
paced lectures. Most of the participants pass course notes and other 
materials on tp their company supervisers and colleagues r use them to > 
prepare memos and presentations to their staff. Others who are currently 
involved in other specialties coin^lain that wheyi they need to review 




the course information at some later date, the materials are ton scanty 
to help them recall the course. Still others need more detail to help 
them on immediate projects. If students are to effectively apply new 
ideas and technology to their work, they need to receive outlines and 
good reference materials in the course. Retention of the mass of data 
presented in the typical short-course is impossible without notes and 
other materials. 

Similarly, copies of the viewgraphs used in lectures are necessary. 
Participants roar if they don't receive copies of all the viewgraphs, 
but also complain if the viewgraph reproductions are inadequately labeled 
or out of sequence. Poor lecture notes and course materials (or the 
lack of them) seem to exacerbate students' feelings of discontent with 
the course and inhibit learning. 

Many engineers would find it very helpful if course materials were 
sent in advance. They say that getting materials ahead of time would 
enable them to be better prepared and ask more relevant questions. 

pending out materials. in advance can be time-consuming and expen- 
sive, but we found that when' instructors did this, participants were 
better satisfied with the course. At a minimtm, instructors could 
include a few references in the course announcement for those engineers 
who are motivated to review and prepare themselves for the course. 
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DISCUSSION 

Evaluation Realities and Difficulties Berkeley Extension routinely 
encourages C£E instructors to give student evaluation forms to each class, 
but like other extension departments, it lacks the resources and dollars to 
perform long-term follow-up studies that might yield information about the 
pay-off of its courses to companies and individuals. Since extension courses 
are expected to pay for themselves, there are limited resources for assess- 
ing and describing the characteristics of the students who attend and for 
assessing the needs and ejqpectations of industries and agencies who send 
engineers to the programs, or for de terming the forinal and informal effects 
of the programs. Books on educational evaluation range from textbooks to 
encyclopedias to do-it-yourself kits. Works by Anderson and Others, 1975; 
Ball and Anderson, 1979; Bloom and Others, 1971; Center, 1979; Dressal, 1976; 
Morris/i 1978; Popham, 1973; 1974, 1975; Scriven, 1974, 1967, 1973; Streuning 
and Guttentag, 1975; and Webb and Sechrest, 1966 illustrate a few of the 
more influential approaches to educational evaluation. 

Extension programs are not exempt from the pressure to evaluate and 
currently books are appearing on how: to assess the impact of continuing 
education courses (Preston and Others, 1979; Knox, 1979-c & d,) Undoubtedly 
as budgets tighten and the economy worsens, there will be increasing pres- 
sure to demonstrate the value of programs. . . and less money to do it with. 

But how widespread is systematic program evaluation in post-secondary 
education? In a survey of practices in program evaluation. Ball and Anderson 
(1975) examined some 200 educational programs divided equally among the 
Department of Defense, other federal government departments and agencies, 



ERLC 



i2 



TAXONOMY OF EDUCATIONAL OUTCOMES 



Educational Development 



Evidence: Level 1 



Level 2 



1. Updates information/increase 
awareness of new developments 
in field. <A,B,C) 

2. Acquires new knowledge 
(A,B,C) 

3. Increases general background 
in subject (A,B,C) 

4. Reviews fundamentals (A,B,C) 

5. Acquires perspective for deci- 
sion-making (A , B , C) 

6. Learns theory (A,B,C) 

7. Gets overview of a new field 
(A,B,C) 

8. Confims previous knowledge/hunches 
(A,B,C) 

9. Gets ideas can use in future- 
(hope-chest) (A,B,C) 

Improved Morale/Sense of Efficacy 
10. Increases confidence in own 
abilites and skills (B,C) 



11. Improves attitude toward job, 
(B,C,D) 

12. Meets and talks with other 
professionals (A,B,C) 



Increased desire to learn 
more about the field (B,D,D) 

Increase receptivity to 
new ideas (B,C,D,E,F) 



Spreads information to others- 
si^ervisor, colleagues, sub- 
ordinates (Ripple effect) 
(C,D,E,F) 



Approaches work with greater 
conf iden je» zest. 
(C,D,E,F) 

Improvt s productivity 
((C,D,E,F) 



Enrolln in other CEE 
cours es , sel f -s tudy 
(C,D) 



Convinces others to 

change . 

(C,D,E,F) 



-n3 



Is advanced or promoted 
in company or gets another 
job. (C,D,E,F) 
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can be useful for ther purposes as well. Students do not object to 
answering basic deinographic questions and skip fewer of them than they 
do attitudinal items. 

Therefore, end-of-course questionnaires are an important evaluation 
tool - provided that they include demographic items, despite their 
sometimes questionable reliability and the biases they may reflect. 
Following up a small sample of students by phone calls or questionnaires 
will yield data on whether the course has helped the individual professiona 
and how she/he has used the information learned. Occassional phone 
interviews with supervisers, company development people, etc. can confirm 
ideas industry holds toward the courses and their perceived effects. 

AN EVALUATION MODEL 

The table on pages 37 and 38 is a taxonmy of educational outcomes 

resulting from participating in a CEE course. The left hand column show.^ 

I 

the changes that students report they gain from completing a course. 
They could also be considered educational goals. There are three types 
of outcomes; educational development, improy^d morale/sense of efficacy, 
and improved job performance. The nine items under educational develop- 
ment range fron: "update information/increase awareness of new developments 
to get ideas can use in future " and represent the kinds cJ knowledge 
acquired • The second category of outcomes are attitudinal— improved 
morale and an increased sense of efficacy . A sense of efficacy refers to 
the engineer's confidence in his own professional competence- the feeling 
that he knows how to do a good job, that he is qualified and productive. 
Efficacy is shown in increased confidence in one's ability and skills, an 
improved attitude toward the job, and the ability to talk with other 
professionals as peers.' 
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estimated at six to seven years. That is, by seven years, half of the 
knowledge the engineer began with has become obselete. Futher, Saxberg 
states, "Theoretically in another six or seven years everything the 
engineer orginally knew would be obsolete. Clearly the only way for the 
professional to combat creeping obsolescence is to continue to move on a 
path of development whereby he maintains and excMeds the job demands as 
these increase or change over time..'* 

Knox {1979-e) discusses the influences on proficiency and obsolescei^e 
as follows; "Some influences on current proficiency and obsolescence 
reflect the indfividuals' s previous capability and backgroxxnd along with his 
current outlook and efforts to be proficient. For example, the currently 
obsolete professional may never have been proficient, or may have acquired 
adequate knowledge but not had the minimum necessary professional experience 
to prod'^ce even minimal proficiency. A professional may forget information 
chat is important to effective practice, or may experience shifts in occupa- 
tional tisks that contribute to obsolescence. Some areas of proficiency 
are not maintained due to lack of practice. A professional's proficiency 
can also be affected by attitudes toward ethical aspects of practice, by 
efforts to interact with peers, by efforts to increase his or her proficiency 
in deliberate ways, by achievement motivation, and by approaches to 
problem solving." 

Because CEE Students who are new to a specialty (even though they 
may have received their engineering degrees a decade or more ago) are 
more critical of instructors and expect to be told what to be told what 
to do on the job Monday rooming, it is important to ask identifying 
demographic questions on course evaluations. This is the only way to 
place the complaints of dissatisfied students in context so that changes 
can be planned and this 
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analysis rather than design synthesis (Meadow, 1977.) Lack of rapport 
between academics and industry has been cited as a reason for low partici- 
pation rates. Also the question of whether CEE courses are properly 
suited to the needs of adults has been raised - although it seems to make 
little difference since CEE courses always have and still continue to use 
lectures as their major teaching method. Busy engineers, like other 
professionals, expect lectures and few seem willing to sign up for hands- 
on laboratory or field experience courses . . . although those who do 
are very satisfied. Our point is not to discard lectures, bui to improve 
them and add mpre time for participants to interact with each other and 
the instructors. 

In our view, company training programs and on-the-job-training seem 
to do a good job of seeing to it that engineers learn what to do and how 
to do it. However, these programs rarely include the "why" and it is in 
this area of understanding that academic faculty can make a contribution. 
We found that enginej.v,( who were very experienced in a specialty were more 
tolerant of the college professors who used hypothetical examples and 
explained theory than were the newcomers to the specialty. New entrants 
seemed still struggling to learn the basic terms and concepts as well as 
hew to do it and were impatient when the "why" was explained. 

The problem of professional obsolescence has been a major issue in 
the CEE literature. (Knox, 1979-e; LeBreton, 1979; Lusterman, 1977; 
Mcuaniel and Others, 1966; Meadow , 1977; Morris, A.J., 1978; Pratt, 1979; 
Rothenbuig 1975; Saxberg, 1979.) Saxberg writes that as engineering 
knowledge expands at a rapid rate, the half-life of the engineer has been 
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^ course with required prerequisities . Giuliani (1979) cites two criteria 

for selecting continuing education programs for impact evaluation: 
1) the extent to which the duration and intensity of a program ccald 
. realistically be expected to produce measurable change in participants or 
their organizations and 2) whether stipport (time^ Mney, evaluation 
expertise r and faculty cooperation) for the evaluation process itself is 
available. 

Brevity and the range of students whom they attract are two factors 
that combine to make evaluation of engineering short-courses particularly 
difficult. The engineers who attend are diverse indeed — from the merely 

0 

curious and those who are just entering the specialty to the professional 
with many years of experience and the researcher whose experiments are 
advancing the frontiers of )cnowledge in the specialty. Instructors usually 
view their goal as helping participants update their skills and knowledge 
in the specialty, but the variety of student needs and expectations and 
their different levels of sophistication pose an instructional dilemma. 
Teaching students who are new to the field and those who are as knowledgeable 
as the lecturer in the same class requires careful planning and great skill. 
Some instructors plan the course with something for everyone in the iinge; 
others try to keep the course at what they c -nsider a basic level; while 
others aim their presentations at a hypothetical middle group somewhere 
between the extremes. 

Because CEE courses serve so may purposes and have such a diverse 
student population, they are difficult to evaluate. Much of the research 
has criticized university-bud^d CEE programs as not addressing the working 
engineer's need for job-rei ted knowledge and too much preoccupation witJi 
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private industrial and commercial enterprises, and junior and community 
colleges. They found that most of the Department of Defense, other federal 
government programs, and private sector industrial and commercial enterprises 
were receiving some form of program evaluation. Most of these evaliiations 
-involved questionnaires administered to teachers and students to measure 
their perceptions of the progtam. However, it was very clear that tradi- 
tional formal education institutions - the colleges - rarely indulge 
in program evaluation in any formal sensii of the term. Ball and Anderson 
state unequivocally: •'It seems that once a program is installed at thm 
college level it becomes 'functionally autonomus', and it is unlikely that 
any formal ef^fort will occur to gather evidence concern;..ng the need for program 
modification or continuation." 

Rigorous, analytical evaluation procedures are infrequently institution- 
alized at thi college level — even wh^»r. . )me sort of eval^^tion is mandated 
as a part of grant funding. A survey of 375 colleges and universities 
that received grants for innovative instructional programs revealed that the 
main pattern was loc .lly developed measures given out by individual faculty 
members. (Hodgkinson, Hurst, and Levine, undated.) 

Evaluating continuing education programs entails the same problems 
that educational ^valuation in general faces, including such important 
factors as the complexity of the process, the number of variables, the im- 
precision of the measures, cost, dislike of people to count results, the 
difficulty of interpreting results, the def ensiveness of instructors and 
progreun directors, and the fact that there are few rewards for evaluation. 
Because tests are rarely given, there are few ways to directly measure the 
amount learned in CEE courses. Nor can one be sure that the students who 
take the courses are adequately prepared in contrast to the typical engineering 

^ 40 



Improved Job Performance 



Evidence: Level 1 



Level 2 



13. Solves present problems 
(A,B,C,D) 




Initiates new projects, 
assumes more responsi- 
bility (C,D,E,F) 



Replicates experiments, 
designs and implements 
new e3q>eriments» (C|D^E|F) 



14. Improves specific job skills/ 
productivity (A,B,C,D,E,F) 



15. Changes procedures/ viewpoint 
(B,C,D,E,F) 



Uses new technology , ideas 
in work (C,D,E,F) 



Code 



Instruments 



A ^ Pre-course questionnaire (for measuring goals 6 ej?)ectations. ) 

B « End-of-course Questionnaire 

C = Follow-up Questionnaire 

D ^ Follow-up Interviews with participants 

E « Follow-up Interviews with supervisers 

F - Follow-up Interviews with colleagues, subordinates 



- Basic formulations may help in my work, but I'm not sure how far I can go without further reading 



future, I'll be using e-beam equipment. Courses like this will help when the time comes to purchase 



- Since I'm new to the field, this course gave me a good background. 

• The course brought me up to date and refreshed my knowledge. 

• The basic ideas from the course substantiated the knowledge I already had and reinforced my decisions to 
ise the techniqv*es. 

- It save me at least 3 months' reading which I probably wouldn't have done. 

- The course gave me new insights into management & design needs. 

- Now I'm qualified to serve new clients and help my company qualify for government contracts, 

- The course definitely increased my range of skills. 

- The information learned will enable me to make faster decisions. 



Response 
Sample Gradient 



/ 
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The third categbry "impro/^d job perfoxrmance** is self-e^lanatory . - 
The engineer reports he can solve present problems, improve his skills and 
productivity, and change procedures. These perceived Qutcomes can be ob- 
served at a later time as illustrated by the items listed in the columns 
labeled Evidence, levels 1 and 2. For exampleS an indiyidual who has 
increased confidence in his own skills approaches his work with greater 
confidence, shows improved productivity, and is promoted or advanced with- 
in his company. The Istters following each item are codes for the type of 
instrument or method used in determing the existence of the itew. 
(See page 38.) 

Context and Constraints 
A n»imber of factors limit the GEE student in using the information 
he learned in a course on his job - indeed from moving to any of the levels 
indicated under Evidence. As Saxberg (1979) states, "The work environ- 
ment sets limits ^>^at training cannot overcome." Meadow (1977) quDtes 
Harold Kaufman as saying that when engineers take a. degree or a coui'se at 
company expense but are not subsequently given more responsibility or 
other form of recognition, their disappointment may cause them to seek 
another job. Similarly, one can leave a course laden with new ideas for im- 
proving one's work, but be constrained because one lacks "rhe power to 
implement change within the company. Some engineers take courses because 
they expect to get into a project relating to that knowledge in the future, 
but are presently so tied into * current project on a different topic 
• that they have no immediate use for the knowledge they gain. Only about 
half of the students we interviewed were able to use the information the^ 
gained in their job,. 
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Personal characteristics (assert iveness, ability, communication 
skills, etc.) also limit an engineer's 2Lbility to apply new information 
learned or to influence others to chajige. 

Long range evidence of transfer of learning is thus dependent on 
many interrelated factors- personal characteristics of the learner, the 
nature of the course (including instructor's objectives, level/pace, 
length, materials, etc. )^ and opportunities « for application in the work 
environment. 

In assessing the impact of short-courses, evaluators must rely on 
self-report measures because achievement tests are neither used - nor are 
they appropriate in* most CEE courses. However, questionnaires and other 
instruments can be improved by asking appropriate questions and by in- 
cluding items that identify key sub-groups (i.e,, those who differ in 
background, experience, expectations, work opportunity to use new infor- 
mation, etc.) Pre-questionnaires as well as end of course and follow-up 
surveys should be administered to at least a sample of students. Follow-,, 
up interviews with participants, supervisers, colleagues, and/or sub- 
ordinates can provide additional evidence of the course's value. 

The Sample Response Gradient shown at the bottom of page 40 
illustrates a way to arrange student responses concerning the. course's 
impact so that they can be weighted as criteriahy independent judges. 
These pafticilar responses have been arramged so that they reflect de- 
grees of learning, but this approach could be refined by adding more 
items and getting experts to weight them. Had we had a longer project, 
this would have been done. 




RECOMMENDATIONS FOR IMPROVING CEE skoRT-COURSES 

Although there are many kinds of CEE courses, we concentrated our re- 
sa'arch efforts on university extension state-of-the-art short-courses that 
do not give credit nor lead to a degree. U.C.B Extension classifies these 
courses as follows: 

1. Courses aimed at updating basic, ongoing knowledge (e.g., 
Airport Planr.ing and Design) or as providing basic training 
in a new technology (e.g.. Interfacing to a Microcomputer) 

2. Regulary scheduled state-of-the-art conferences 

(e.g., Computer Aids in IC Technology and Device Design) 

3. Special one-F^ot topics of interest to a special population 
(e.g,, Electron-Beeun lithography) 

From the students' perspective these goals overlap so that, whatever 
the coordinator's intention, the same course may attract students seeking 
to fill each of these needs. 

The recommendations that follow apply specifically to university 
extension courses, but may have implications for programs offered by companies 
and professional organizations as well. 

However, university extension programs operate under special constraints 
that may\differ from those of other providers. As appendages of universities, 
extension CEE programs must maintain credibility with their engineering faculty 
as well as with the industrial community. U.C.B. Extension's goal is to pro- 
vide a hi9h quality public service, but, at the same time, prograuns are ej'- 
pected to pay for themselve.^. Thus economic realities liirdt the kinds ol pro- 
grams offered and determine how often they are given. Futhermore, extension 
programs compete with commercial programs that don't need the approval of 
engineering departmc^nts and can concentrate on profit making. 
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U.C.B. Extension CEE courses aim at bringing faculty who work at cutting 
edge of new knowledge together with engineering professionals and praction- 
ers* The fact that every course is expected to be a winner and pay its 
own way is typical of extension progreuns throughout the nation, (Meadow 1977) 
At U.C.B. CEE course failures are few^ but costly when they happen. However, 
some programs draw large numbers of participants en£QDling the department to 
offer a few classes that attract small audiences. 

The problem of maintaining the academic standards of graduate engineering 
faculty is an unending struggle, however each course is supe^-vised by a 
faculty-member-in-charge, and a board of overseers comprised of engineering 
faculty and industrial representatives advises the CEE administrators. 

The U.C. Berkeley CEE program has one advantage • that is, the prestige 
of the university and its research thrust attract mcmy engineer practitioners 
to its courses. 

So, taking into consideration the special characteristics of UCB CEE pro- 
grams, let us examine the implications of our findings for improving courses 
may useful to other 'progreur^s. 

Roles and Responsioilities of the CEE Program Director , 

The Program Director is the key to the development of successful CEE 
courses. Directors, should be dynamic, knowledgeable, and effective in working 
with engineers for they must continually interface with industrial leaders 
as well as academicians. They must identify industry's needs and locate 
and convince the best quaiifiea instructors to teach CEE courses. 

Thev must be dilligent talent scouts for they often have to scrounge to 
get good instructors. They must persuade engineering faculty members that 
Extension teaching is a legitimate function of tie university and overcome 
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the negative attitudes some academicians hold about teaching CEE courses. 
Also industrial leaders need to be convinced that university professors can 
teach practicing engineers effectively. Some CEE experts claim that pro- 
fessors are not the best instructors for practicing engineers because they 
usually lack industrial experience and an understanding of the realities 
of engineering jobs (Meadow, 1977) , Professors have also been criticized 
as hot adequaltely prepared to teach state-of-the-art CEE courses by those 
wh(0 credit industrial researchers with producing most of the significant 
applied research. Undergraduate textbooks and courses are often ten years 
behind the time and journal information lags behind current resaarch by 
several years. 

However, in a prestigious institution like University of California, 
Berkeley, the greater problem is how to make CEE teaching attractive to 
professors who have state-of-the-art information that would be useful to 
practicing engineers. Extension teaching is viewed as a third-rate endeavor 
for faculty who consider teaching and research their highest priority 
and rank undergraduate teaching as second. 

Locating good instructors from the industrial ^♦arch community is 
not easy either. Finding an engineer who is open enough to share information 
about current research without irunning the risk of revealing company secrets 
is difficult. (Even when speakers are open to discussing their research, 
students remain unconvinced when they say that the problems are still unsolved, 
preferring to believe that the speaker's company has solved the problem, but 
does not want their com^petition to know about it.) 
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Program directors must keep th^^ .r fingers on the pulse of industry 
and know what courses are needed. It would help to be clairvoyant so that 
one can predict what information and skills engineers will need in the near 
future. Short of this, directors must maintain personal contacts with 
industrial leaders and know how to collect needs-assessment data quickly 
and cheaply. (Rarely do program directors have the time and money even 
if they had the inclination to perform large-scale formal needs-assessment 
surveys.) Directors must keep up with the areas in which knowledge is 
growing rapidlySo that they can anticipate technological changes. 

Once they have identified potential topics and instructors, the 
director must help the instructor decide on how to package the information 
into courses that will be ""acceptable both the engineering faculty and 
industry. The packaging should serve a more rational purpose than Extension's 
need to fill courses. In some cases, we observed courses that covered 
several specialties, and found that when participants had narrowly limited 
interests they complained their interest was given short-shrift in the course. 

CEE Directors should help instructors plan their courses, give them 
information about the probable characteristics of their adult students and 
describe the pitfalls to avoid. 

For example, the CEE Director should monitor course titles, course 
descriptions, the schedule, as well as the course brochure and other publi- 
city . 

Course titles should be clear and accurate. Calling a CEE course, - 
"Fundamentals of . . . . " will draw many students who have no background in 
the specialty, if the instructor aims his presentation at fie hypothetical 
average student (somewhere between the novice ar.a the state-of-the-art 
researchers), his presentations may be over the heads of most of the class. 
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A course covering the same content, speakers, and level could be called 
"An Overview of and not discourage as many students even though 

some ine)q>erienced people will enroll. Or if the instructor's aim is to 
present state-o^-the-art information, the course could state that "only 
enough fundamental will be covered to help students understand new develop- 
ments in the field." 

If two topics are to be covered in the course, the title should accura- 
tely reflect the emphasis on each. For exaunpla, if the title is "Planning 
and Design* , both planners and designers will attend and may expect equal time 
and or equal emphasis. Dissatisfaction arises when the designers for 
example, feel the course has not given them the information they expected. 
One way this problem might be^ eased is to permit the designers , for example, 
to attend only those sessions of interest. Another way is to plan the 
course so that design information is an integral part of most or all of the 
presentations. If the latter strategy is followed, it should be clearly 
described in t^ie course announcement. 

Course descriptions should be as clear and as specific as possilDle and 
accurately reflect course content . 

It seems that more care, effort, time and w^oxTars go into designinQ the 
artwork, colors, and layout of the brochure that advertises the course than 
in writing ttie course description or even in preparing the course materials. 

The course description must bo carefully worded, and some instructors 
may need more editorial help for Extension than they presently get. At 
a minimum, the course description should include the following: 1) the 
topics to be covered, 2) the rationale (Why should a student take this 
course?), 3) the expected outcomes or goals (What does the instructor expect 
to achieve?) , 4) a description of the level of experience or background 

So 
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expect:*.d of students, (If the instructor assumes' prerequisite knowledge 

•r 

this should be clearly stated), 5) a description of the instructional 
materials (textbooks, manuals, etc.), and 6) a statement about instruc- 
tional techniques (lectures, discussion groi?>s, question and answer 
panels, etc.) 

If course materials are not sent to participants in advance of the 
course, ic would be very helpful to list a few basic references in the 
announcement so that engineers who want to prepare themselves for the course 
may do so. 

It is a good idea to include the class schedule in the brochure 
and the prograun director should see that it is not too crowded with topics, 
that sessions are not too long, that aunple time is allowed for discussion, 
that breaks are long enough (15 minutes is too short) , and that lunch periods 
permit participants time to relax a little, e?.t, talk and digest their food. 
(In other worths, don't show technical films during luTiCh hour.) 

Students in short-courses often suffer from information-overload and 

may complain when they feel sessions are too long. Shorter, mCre specific, 
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well-organized formal presentations with more time allowed for discussion would 
in our opinion, maximize the students' ability to absorb, retain, and transfer 
the course information. 

One practical problem in preparing course descriptions and schedules 
is that the brochures are printed four t-:^ months before the courses are 
given. I suspect that instructors sometimes forget what they have written 
between the time the brochure goes to press and the week before the class 
starts when they pull th -i final product together. Although speakers 
and even the schedule may need to be changed, this should not be an excuse 
for vague, cursory course descriptions. The course goals, topics, should 
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remain essentially the same regardless of changes in speakers and times. 
Course Materials Engineers who taU^e CEE short-courses need and expect to 
received complete outlines of the presentations. In information-intensive 
courses it is a practical impossibility for many engineers to take adequate 
notes. Ph.D's complained more often about the lack of lecture notes in 
courses where they were not distributed than did engineers with other 
types of degrees. Copies of viewgraphs and other a/v materials are also 
eaqpected in the materials given out in the class, (See discussion on page 
27.) 

D^eloping good comprehensive courses notes and other materials 
for students requires a great deal of time and effort to write, edit, and 
reproduce. This is an area in which Extension could help by providin5 
instructors with nvore technical writing and editing assistance. Also in- 
structors in CEE courses must update their materials each time the course 
is offered. Instructors may need additional incentives to do this. 

At a minimum, course materials should include detailed notes of "the 
main lectures in hard copy, copies of viewgraphs that are labeled, other 
articles or research material with their relevance to the course explained 
clearly, and bibliographic references. 

Because enginieers often enroll in CEE courses to get the materials, 
improving the materials would pay-off in increased student satisfaction, 
increased clientele, and an inproved probability that the participants 
would be able to apply course concepts and technology to their work. 

CEE instructors seldom send out course materials to students before 
the class begins, in fact, the materials are usually pull.-^ together at the 
last moment. There does seem to be some justification for encouraging in- 
structors to send materials in advance, particularly in highly specialized, 
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technical, one-day programs. Students were more favorably disposed to the 
subject in course where this was done and a number of engineers requested this 

in courses where it was not done. 
HELPING INSTRUCTORS DEAL WITH STUDENT DIVERSITY * 

Anothej^ important function of the CEE Director is to help instructors 
understand the special problems in teaching adults and help them avoid some 
of the pitfalls that can occur. 

Tie range of preparation of students in a CEE class poses problems 
for engineering professors who are accustomed to teaching their specialty to 
relatively homogeneous classes in a tight, sequential curriculum. A CEE 
class may seem to include the equivalent of a herd of sophomores, a gaggle 
of jurfior and senior engineers, a handful of the professor's best graduate 
students, and a few people who walk off the street. Each student has a unique 
perception of the course and different practical needs. 

In planning state-of-the-art one-day courses, experienced CEE instructors 
usually build omnibus programs that sacrifice depth for breadth and include 
something for every level-that is, some fundamentals, a bit of theory, current 
research findings, practical examples and applications, and speculation on 
future developments. Speakers are chosen to present each of these Industrial 
representatives teach along with engineering professors. Inevitably the 
speaker who presents the theoretical material, regardless of his or her sp^J^ing 
prowess, earns lower student ratings than the lecturer who describes appli- 
cations. The overly-theoretical college professor is thus as at disadvantage 
in CEE courses, but so is the speaker from industry who confines his examples 
to one process or product when the class is comprised of people working in 
different industries or with different products. 
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Large state-of-the-art one-day courses attract diverse stud^^nts, but only 
about one-fifth are new to the subject.. Thay also provide a variety of 
spj5aker3 arc topics to meet participants^ varied needs. Serious problems arise 
only when the schedule goes awry and some topics are not covered as thoroughly 
as the participants anticipated or when speakers are poorly prepared, inade- 
quate in delivery or present purely theoretical or mathematical talks. 

We found that instructors had the greatest difficulty teaching smaller 
classes (under 50 students) when students varied greatly in background 

and experience and when about half of the class were novices in the field. 
The courses that students rated the best and where there was more, evidence 
that participants transferred the information from the class to tlisir work 
were highly specialized small (enrollments of under 20) with relatively 
homogenous students. For exanple, participants in the Value Engineering course 
teneded to be older, highly experienced engineers working in design and 
construction. Smilarly the microproc':issor course drew people with previous 
experience with computer programming. 

As we have mentioned before, there are things that instructors. can do to 
reduce the diversity of students in their CEE courses - such as making sure 
that the course title reflects the course content, and clearly describing in 
the course brochure the level of the presentations and the prerequisite know- 
ledge and experience needed to understand the course in the course brochure. 

However, if one cannot restrict enrollment to those who might profit 
most from the course, it would always help if students answered a few qut:stions 
about themselves be fore the course begins. With these answers there would be 
fewer surprises. Instructors would have some idea of tlie students' backgrounds 
and interests. A simple pre-questionnaire like this one could be printed on 
tne brochure and sent in with the registration form: 
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Pre-Questionnaire 

1, Briefly describe your present position* 

2, Is your work primarily managerial technical other 

3, What are your reasons for taking this course? 

4, Have you taXen other courses in (course topic) or related courses? 
If yes, what where and when? 

5, Does year present work involve (course topic?) If so, how? 

6, How long have you worked in your present specialty? 

7, What is your highest degree? When did you receive it? 

8, What do you expect to learn in this course? 

Instructors should be encouraged to try different teaching methods to 
better meet the needs of adult students* CEE courses are rarely sequential- 
that is, there are usually not separate courses for beginning and advanced 
students and everyone,- regardless of background and experience, enrolls in the 
same course. Therefor^^ it is important that ^hese differences be recognized 
within the class by providing optional sessions, more discussion groxjps in 
which experienced practitioners could share their practical knowledge, adding 
on a hands-on lab for newcomers, or an extra review session on fundamentals. 
Instructors seem reluctant to try other teaching models other thain lecture/ 
discussion. To deal with the varying adult learning styles and capabilities, 
an eclectic approach to teaching is needed. That does not mean that lectures 
should be eliminated - they are essential and a good way to convey information 
quickly to a large number of people. Engineers, like other professionals, 
do not want to waste time, and prefer good, well-organized lectures. However, 
there are practical limits as to how much can be absorbed from a full day 
or a full week of listening to lectures. 
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Holding concurrent sessions during a course where students from different 
specialties could chose to attend one or the other has not been tried in 
the courses' we observed. Everyone sits through all the presentations. 

Another method for insuring that students transfer skills and .knowledge 

is to plan brief follow-up sessions - e.g., 4 to 6 weeks after the original 

course. This would give instructors a change to provide feedback to students 

• a condition that maximizes tranfer. (Guiliani, 1979) i 
Additional CEE courses are needed * i.e., courses that engineers 

can take after they are turned on to learning more about a topic 

by a one-day course. It is vital that general courses be followed by 

intensive programs that provide hands-on work or more advanced concepts. 

This is a need that is not being currently addressed by Extension. 

Evaluating Course s The CEE Director also has a responsibility for evaluating 
courses on a more sophisticated basis than whether the course enrollment 
fills and the dean receives no letters of complaint from students. . Without 
some additional evidence of a courser's value, few risks are taken - that 
is, if the course filled, it is offered again without change. 

Compiling a sin5)le data base (such as that shown in Appendix B) Is a 
first step in gathering information for evaluation. Designing questionnaires 
for end -of -class administration, and an occasional follow-up of a random 
sample of students are other ways. People do not like to count and analyzing 
course questionnaires does take some clerical time. As we have shown, it 
isn't the a£)solute number of complaints that instruction is not meeting engineers 
needs that makes the difference. But rather, instructors seem 

unaware of the number of inexperienced students in their classes and 
novices are the ones most likely to complain that the subject matter 
and presentation are over their heads. 
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Changing Industry's Views About (S^E Gathering simple data on the value 
of courses can help fill another need - that is^ the CEE Director must 
assume a missionary role and convince industrial representatives to change^ 
their attitudes about CEE. Students who have completed courses need to 
be offered opportunities to take itiore responsibility, and challenged to use 
the information and techniques learned. 

Somehow the attitude that taking CEE courses is something one-' does 
on one's own if the company pays - like a fringe benefit - needs to be 
changed. With adequate evidence of the value of CEE course work, a direc- 
tor might be aO^le to convince companies t lat they too can benefit if they 
encourage engineers to apply new information. 

In suninary, the CEE Director serves many roles and devisir4g simple 
but effective ways of evaluating the programs is one of the most important. 
Unanswered Questions In this project we have looked at participants' 
reactions in 10 CEE short-courses and used these data to develop 
a simple model for evaluating" CEK short-courses. A spin-off of this 
effort has been a number of suggestions for directors and instructors ^-^ 
given in the preceding section. 

Time and money limited thejamount of follow-up and analysis that ^ve 
could do. We did find a number of questions that we were unable to answer, 
and these might aid other investigators in further investigations: 

1. Taking a CEE course seems to increase participant's interest in 
t5Jcing additional courses or further study. How long does this enthusiasm 
last? Do engineers continue to take courses or do they find other ways 

to fill this need? 

2. How often do engineers need updating? In the courses we examined 
some people come back every year, others every two or three years. Certainly 
attendruice at some of these courses depends on how rapidly the technology 
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in the field iz changing, but/ aside from that, are there any yard-sticks 
for predicting when one should return for a refresher? 

3* Do engineers who complete CEE courses transfer jobs more fre- 
quently than those who don't take courses? We found that the ability to 
at>ply information learned on the job is a function of the job setting- 
and company constraints. If an engineer returns to work desiring to try 
something new and is thwarted - what happens? 

4, It was surprising to us that- very brief one-day progic^ms resulted 
in chamges in work and transfer of ''training with some participants* what 
are the factors that make this possible? Are they all job-related or are 
some related to the nature of the individual? Are there ways of deter- 
mining who will profit from a CEE course? Are there ways of packaging ^r 
advertising the courses that would attract engineers who presently do not 
attend CEE courses? 

5. How might courses be^^esigned to maximize the chances of an 
engineer spreading the information to others in his/her company? Are there 
ways companies could encourage this? 

These are some of the questions that still intrig\ie us. 
To summarize: In this project we developed a simple model for eval- 
uating CEE short-courses, based on the problems, processes, and findings 
of 10 evaluations of U,C,B, CEE courses. The model developed addresses 
three major areas of outcomes: educational development, improved morale/ 
sense of efficacy, and improved job performance. Under these three 
categories are 15 specific outcomes and two types of evidence for classi- 
fying results. A gradient of statements reflecting the impact resulting 
from CEl education was sketched. To place the outcome evidence in appropriate 
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context, it w^is suggested tha^ characteristics of the job setting and 
personal characteristics of the participant be considered. 

Routine evaluation of CEE courses can be done inexpensively by 
improving end of course Questionnaires and including some demographic 
questions (i.e., it is important to know the student's reasons for taking 
the course, the amount of experience he/she has had in the specialty, 
and background in the subject, etc.) Compilation of a simple data base 
from this infoiination will ease planning. In our study we found^ for 
exiunple, that engineers who were ine)5)eriencQd in the specialty were 
more likely to con^lain that instructors^ put too little emphasis on 
applications. These engineers although newcomers to the specially tended 
to be older engineers who were considering or had just made job changes. 
Perhaps much of the concern about job obsolescence has been mis-directed 
by over-emphasizing the engineer who remains in the same specialty 
throughout his career. It may be more realistic to recognize that many 
engineers are forced into mid-career chaoiges and given limited time to 
prepare for new specialties. This group is rarely addressed dirtictly in 
the planning of CEE courses, and perhaps some of the suggestions we have 
made for improving CEE courses will aid their plight. 

Dissemination of this Report . A copy of this report will be sent to the 
ERIC Clearinghouse for Continuing Education and copies will be sent to 
75 libraries. In addition, copies of the abstract will be sent to 100 
industrial training representatives. 
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APPENDIX A 
INSTRUMENTS USED IN STUDY 



STUDENT QUESTIONNAIRE 
ITEMS 



The National Science Foundation has funded U.C.B. Extension to evalu- 
ate the quality and usefulness of its engineering short courses- Your 
responses to this questionnaire will help us improve our courses and 
develop an evaluation model for other courses. 

Please fill out the following items carefully. Most questions can be 
checked, but a few require short answeres. 

We appreciate your taking the time to answer these questions. The 
informaMon you give will be held confidential unless you agree to 
release it. 



I do ( ) do not ( ) wish the information I give to be released to 
my company. 

(Note: We found that the more students answered the questions when 
the release statement was at the end of the questionnaire.) 

Democraphic items 

What is your job title? Your highest degree? 

In what year did you receive it (or year)? 

What % of your time is spent in: 
( ) managerial duties 
' ) technical duties 
{ ) sales/marketing 
( ) other (what?) 

(Nc i'3: other alternatives used in this item varied with the course 
including: test/quality assurance, production, design, structural 
engineering, research, production processes, contracting.) 



Do you 'rfork primarily in: 

1 ; R & D 

( ) Production 

) Planning and Oecision-making 

(rote: other alternatives were used depending on the course.) 



Demographic items (continued) 



How long have you been in 7 (lithography or whatever thf course 

topic is,) 

(Revised to: How many years has your work involved earthquake engineer- 
ing , or whatever the course topic is?) 

(For some of the computer courses we asked questions like?) 



How much experience have you had in using computer analysis techniques 
for designing buildings? 

)• Have written applicable computer programs • 
) Have performed computer analyses. 

) Have used and interpreted the results of computer analyses. 
) Have had other experience, Explain 
) Have had no experience 

(for value engineering we asked: Are you employed by the goverment? 
How many people do you supervise? 

How much opportunity is there for you to disseminate the information 
you learned in this course to other in your company? 



How well prepared were you for this course? or 

« 

How many courses in have you taken? or 

Have you taken this course (or one like it) before? 
If so, where and when? 
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EVALUATION OF COURSE 



What were your goals in taking this course? Rank order 
1= most important 
4» least important 



learning the fundamentals 
improving general background 
updating knowledge 

acquiring a perspective for decision-making 

meeting others in the field and finding out what they're doing 
other (what?) 



How well were your goals achieved? 



Learning fundamentals 

Updating knowledge 

Acquiring perspective for 
deci sion-making 



meeting others 
other (what?) 



not at 
all 



) 



some- 
what 



satis- 
factorily 



very 
well 



(Note: on some course/questionnaires the goal questions were combined.) 



What were your goals in taking this course? 
How well were they achieved? 

not at all 



somewhat 



very well 



) updauing technical information 
) knowledge of fundamentals 



) 



perspective for decision-making 

) meeting others in the field and 
finding out what they're doing 

) improving job skills 



i^Note: on some questionnaires we included "getting an overview of current 
research/' "learning advanced theory," "Solving process problems," etc. 
at the instructor's suggestion.) 
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Evaluation of course continued 

How did this course ccmpare with other engineering short courses you 
have taken? (In usefulness^ contents interest) 

( ) the best I have taken 

( ) better than others 

( ) about the same as others 

( ) less satisfactory than others 

( ) have taken no others 

Rate the course on the following: 

P^ce ( ) too fast ( ) about right ( ) too slow 

length ( ) too long ( ) about right ( ) too short 

difficulty ( ) too hard ( ) about right ' ( ) too easy 

How would you rate the instructors' emphasis on the following? 

too mujh about right too little 
current research information ( ) ( ) ( ) 

fujridameirtals ( ) ( ) ( ^ 

application ( ) ( ) ( ) 

theory ( ) ( ) ( ) 



Was there something you expected to get from this course ^ but did'nt? 
if so, what? 
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Evaluation of Course continued 



How effective were the various forms of instruction? 

poor fair good 
team workshops ( ) ( ) ( ) ~ 



excellent 



films 



( ) 



lecturt a 



( ) 



Was there something that you expected to learn in the :ourse, 
but didn't? If so what? 



How might this course be improved? 



Outcomes and Applications 

Do you plan to do further study on the topics in this course? 

( ) yes, as part of my job 

( ) yes, on my own time 

( ) doubt that I'll do more study 

( ) am sure I won ' t 



Have you sought further instruction or done further individual or company 
research on the topics in this course? 

( ) major company research area 



OR 



have done considerable research on my own time 
have taken additional courses 



no 
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Outcomes and Applications continued 



How useful have the lecture notes and/or other courses materials been 
to you 



( ) very valuable 



; { ) quite useful 



( ) not useful 



Which phases of the vorkshop will be post useful to you in your work? 
either immediately or in the long-term? 



Of immediate us e 



Of future use 



Information 

Speculation 

Analysis 

Development 

Presentation 

(Note: Topics varied with the course •) 

(On l:he reverse side of this pa^je, briefly describe the ideas & techniques 
you found most valuable, and how you plan to use them.) 



OR 



Did you find anything in the 
•course that you could use 
immediately? if so, what? 



Did you find anything in the 
course that you can use in the 
futre? If so, what? 



TABS 
ETABS 
DRAINE 2D 
DRAIN- TABS 

(Topics varied with the course) 

{On the reverse side, briefly describe the ideas and techniqufjs you found 
most valuable and how you used them.) 
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Have you discussed the information gained in this course with 
your company colleagues ( ) supervisor ( ) 

Do you think that taking -this course has helped you professionally? 
( ) yes ( ) no ( ) 2un not sure 

If yes, explain: 



How much did your company benefit from sending you to this course? 
( ) a great dea] ( ) somewhat ' ( ) not at all 

Explain: 

Would you recommend this course to a colleague? ( ) yes ( ) no 
Company Support of CEE Courses 

Who initiated your decision to attend this course? ^ 

( ) I did \ ) my superviser ( ) i was selected 

by my company 



How does your company support/recognize attendance at courses like this 
one? 

( ) conpany strongly encourages attenda;ace 

( ) company pays fees and expenses '( Full- Partial- ) 

( ) company grants released time (time off) 

( ) course attendance increases possibility of promotion/ raises 

( ) no particular encouragement 
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I would like to participate in a short course on: 



Other remarks? 



Final Statement 



The lisy Evaluation Project Team is planning to conduct case-studies on 
engineers who have taken short-courses. If you are willing to participate 
in, two brief interviews, to be scheduled at your convenience , and to 
perroit us to interview your colleagues and supervisor, please fill in your 
name and phone number below: 



Naime 



Phone No. 



(area code) 

( ) perfer telephone interview 
( ) want further information 
( ) do not wish to participate 



( ) I'd like a summary of the final report. 



Please return questionnaire to: 
Ergr. Extension #XT37 

2223 Fulton, Berkeley, CA 94720 



A- 9 

PARTICIPANT INTERVIEW DATA SHEET 
Course Name 



Code # 



Date Interviewed Phone 

Interviewer job title 



How do*s your job involve thf.- topics covered in this course? 
Why did you take the course? 

Hov; much did you know about the topics beforehand? 
How do you feel about your icr.owledc^ now? 

Have you had a chance to use information learned in the course? 

If 30, hOvV? 

Have you used the course notes and oth:=r materials? How often? 
For what purposes? 

How many people do you supervise? Has there been a chance for you 
to spread information from the course? 

Describe ^^^^ changes as a result of taking the course 

Have you taken any related courses since taking this one? 
If 30, what? Had ycu had any before this one? 

Where else could you get instruction on these trpics? 
Was the course oriented toward practicing engineers? 

^^^t^^^"^ "^y-''^ ^^^^ instructors presented the 

class? oould you corr.rrtent on specific instructors? P^^^^^^^^ ^^e 

'Are there better wayu to run a class like this? 

Are there any other topics you'd like Extension to present? 

AnyonrL^you^d^parC^^^ education? 

co-Lses or^giv^u^'^'n^L^^'ofnoi'L^h'^^o^r^f^^^^^^ 

from having you take this course? company has gained 

If so , names +^ +i « / ^ 

^title(s)_ 



phone #'s 



-e anything else you'd like to add? 
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FA -Z ULTY INTERVIEW PROTOCOL 

/ 

Ins txuc tor's name ^Course_ 

1. Would you explain the* specialty (content) the dourse addresses? 

2. What other sourceji of training are there in this specialty? 
(i.e.# are courses included in undergraduate/graduate engineering 

fichools? company training programs? others? If so, how long do these courses 
last? HOW do they differ from this CEE course?) 

3. What do you see as the major goals of this course? 

4. What kinds of students do you expect to attend (Background prepara- 
tion, reasons for enrolling? Engineers? Others?) 

5. What do you estimate the educational background of the course partici- 
pants to be? (% BS, %MS, % PhD? Other?) 

6. Have y ; .\ taught this information as a CEE course before? grad/undergrad 
course? ' rv 

7. Do you . .;: this course toward any particular educational or experience 
level? 

8. How was "ho course selected? 

How did ycM decide on the teaching methods used in the course? 

10. K")w did yo select speakers? 

11. What kinc's of problems do you expect students to have in learning the 
course material? 

12. Will cour5e notes or other materials be given to the participants? If 
so, please describe them 

13. What Kinds of effects do you expect the course to have on those who 
takan it? (Examples of transfer.) 

(Other questions were tailored to each course. For example, we asked the Value 
Engineering instructor about tJhe importance of certification? how companies 
recognized CEE participation; whether government employees would enroll, etc.) 
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NSF Proj. XT-37 
M. Maxwell 



Follow-up Interview Form (for superviser, colleague, or subordinate 
Name Phone Company Date 



Interviewed by 



Code # 



1. Introduce self and explain purpose of NSF Project 



2. Explain that 



has consented to let us 



(name of student) 
interview you regarding the impact of CEE courses on the company. 
We're interested in how the company gains from sending representatives 
to CEE courses and what impact these courses have on employee's job. 



3» In general, new do \ou feel the company benefits from sending 
representatives to CEE courses? 



4. 'jVhat effects have you observed from 



attending the course? 



Examples 



code: changes in work productivity 

procedures 
persepctive 
techniques 
. work with clients 
work with colleagues 
other: 

problem- solving 



5. Has 



Examples : 



6 • Has 



Examples ; 



implemented anything new? 

programs 
res^ajf'ch 
dec IS ion- making 

procedures 
other? 

disseminated information learned in course to others? 

who? superviser, coll. , subord. 
how? passed around ^materials/memos 
presented inf o • in meetings 
discussed informally 
informal discussions/brain- 
stormed 

*s professional development? 



?• Do you feel course contributed to 
If so, how? 



increased skills in 

understanding of^ 
confidence in 

S leadership in 
> job skills in 
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Interview Form (superviser, etc.) 



8. Other specific benefits to company? 



9. Any negative spin-offs- of taking GEE courses? 



10. Suggestions for needed courses/ ways courses could be improved? 



So 
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UNIVKKSITY OK CAl-IFOKMA, liblHKKLKY 



hl'HkEl.h\ • UVVl'i ' iHM.Nfc • A.Nt;H * lUVtHMnK • SAN IMKCO • NAS IKANCISCO 




SANTa UAKUARA ♦ SA.VTA cnu7. 



t'MVFHSlt"^ KXTKNsUlV llliiJ KULTON STRKKT 

BERKELEY, CALIFORNIA 94720 



June 11, 1979 



TO: Students in the 1978 Airport Systems Planning and Design Course 



[in^ineer iiig Extension, under a grant from the National Science Foundation, 
is conduct in)? an evaluation of its short-courses for engineers. Your responses 
to the questions on the attached questionnaire will help us find ways to improve 
our courses and to develop a model for e/aluating future courses. We are par- 
ticularly interested in finding out whether and how you have applied ideas and 
technique J learned in the course to your wck. 

Your responses to this survey will be kept confidential. •However, we have 
included your name on the questionnaire so that we will not have to bother you 
with reminder letters. 

You may request a summary of the evaluation results by checking the box at 
the end of the questionnaire. 

Please take a few minutes to answer the questions and return the form in 

the enclosed, pre-addressed envelope. 

If you have any questions about the survey, please feel free to call me 
at i4l5] ::3-5947, 

rhank you for your help, 

m 

Sincerely yours, 

MARTHA MAXWELL, Ph,D, 
N,S,F. Project Director 

P.S. We are also asking for volunteers for cace-studies of the impact of 
Extension courses. If you are willing to be interviewed by the project team, 
and permit the team to interview your supervisor and colleagues, please in- 
dicate this on the questionnaire. 



So 
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DATA BASE (406 Respondents) 

I. Student Characteristics 

1. Educational Background 

High School - 1% 
Assoc. Deg. 
Some college 6 
as 45 
MS 30 
PH D 18 

2. Year received highest degree. 

78-9 9% 
76-7 13 
70-5 31 
65-9 19 
60-4 11 
59 or earlier 18 

3. Percent of time spent administration: 

0% 25% 

10-29 31 
30-39 9 
40-49 6 

50-69 16 
70-89 8 
90-100 5 

4. Percent of Time in Technical duties: 



0% 




10-29 


13 


30-39 


4 


40-49 


4 


50-69 


19 


70-89 


17 


90-100 


41 



Note: Not all participants answered all questions and the data 

presented are not weighted by class size. Therefore, this 
table serr/Bs only as a general model of kinds of information 
one might compile in a data base. 
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Number of people supervised: 



None I2^i 

1-2 16 

3-4 29 

5-10 40 

10 or more 9 



Number of years in field: 

Less than 1 year 23% 

1-2 22 

3-5 15 

5-10 12 

10 or more 28 

Who initiated your taking the course? 

Sent by company 10% 
Supervisor 

Recommended 26 

Self-initiated 63 



Company support of CEE course 
attendance: 

Co-^pany pays fees & expenses 95% 
O .apany gives time off 93 
Company strongly encourages 
attendance: 82 

Previous courses on this topic? 

Yes - 26 % 
No - 74 % 

Previous CEE short courses? 

Yes - 66% 
No - 33% 
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II STUDENT EVALUATIONS 



1. Degree to which goals in taking course were 
satisfied: 

Goal - u{ iate knowledge (% checking this as goal 
= 75%) 

Satisfaction: 

Not at all 5% 
Somewhat' 40% 
Satisfactory 15% 
Very well 40% 

Goal • learn fundamentals (% checking this as 
goal * 67%) 

Not at all 6% 
Somewhat 43% 
Satisfactory 17% 
Very well 34% 

Goal - Improve general background 

Not at all 6% 
Somewhat 29% 
Satisfactory 31% 
Very well 35% 

Goal Solve Problems 

Not at all 19% 
Somewhat 64% 
Very well 17% 

Goal - Acquire perspective for decision- 
making (65%) 

Not at all 13% 
Somewhat 47% 
Satisfactory 12% 
Very well 28% 

Goal - Meeting others in the field (55%) 

Not at all 18% 
Somewhat 51% 
Satisfactory 13% 
Very well 19% 



ERIC 



p. B-4 



Goal - Improve job capabilities (10%) 

Not at all 15% 
Somewhat 47% 
Satisfactory 7% 
Very well 31% 



Goal - New Field 

Not at all 13% 

Somewhat 18% 

Satisfactory 19% 

Very well 50% 



2. Comparision with other CEZ courses 
taken : 



ERJC 



Best' 


7% 


Better than 




others 


30% 


About the 




samei 


25% 


worse 


6% 


Hive taken 




no others 


33% 



Usefulness of course material: 

Doubt that I'll use then; 10% 
May use them 37% 
Expect to use them regularly 53% 
(Very valuable) 



4 . Outcomes : 

Interest in taking other courses: (write-in) 
Yes 45% 

Plan to take furthei^ courses 
Yes 72% 

Plan tc do further research on course 
topics on own? 
Yes 77% 

Plan to do further resecirch on job? 
Yes 75% 



B-4 Continued 



5. Application of Information learned in courses; 
No. Topics/ideas can use immediately 

0- 39% 

1- 25% 

2- 15% 
3-4-14% 

5+-5% 

No. Topics expect to use in futxire 

0- 45% 

1- 23% 

2- 17% 
3-4-10% 

■ 5+ -5% 

No. examples given: 

None-69% 
one-19% 
twp-8% 
three-2% 
four, or or-more 2% 



82 
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6. Was there something you expected to get 
from the course, but did'nt? 

Yes - 58% 
No - 42% 



7. Evaluation of aspect of course: 



Pace 

Length 

Level 



Too Fast 

13% 
Too Long 

13% 
Too Hard 

14% 



About Right 

81% 
About Right 

55% 

About Right 
66% 



Too slow 
7% 

Too slow 

32% 
Too Easy 

20% 



Insuructors ' Emphasis on: 

Too Much 
Research' 5 

* Applications 2 

Theory 13 

Fundamentals 8 ' 



About Right Too little 
74 21 



59 
75 
81 



40 

12 
12 



8, Written suggestions for improving the course: 
(Based bn 217 comments) 



Improve materials 

Organization 

Instructors 

Con tent/Topics 

Focus more^on 
Engrs. needs 

Facilities 



27%. 
22% 
15% 
13% 

11% 
8% 



Other 



4% 
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III. Other Items: 

1. How well prepared for this course weije you? 
Unprepared 13% 
Somewhat 26% 
General Background 24% 

I . 

Well Prepared 35% 
. Very Well Prepared 3% 

2. Willing to be interviewed ? 22% 

Were interviewed 15% 

(Note: 68% of those willing to be interviewed, 
were interviewed) 



APPENDIX C 

EVALUATION REPORTS ON INDIVIDUAL 
CEE COURSES 

1 

Course ^ Page Number 

Pundainentals of High Resolution Lithography c-1 

Composite Materials Computation Workshop C-2 

Valtif Engineered Design and Construction C-3 

Electron-Beam Lithography C-4 

Airport Systems and Design C-5 

Exponential Smoothing and Adaptive Forecasting .C-S 

Eartl.quake Analysis of Multi Story Frame & Shearwall 

Buildings C-7 

Engineering Design in Timber , C-8 

Computer Aids for IC Technolgy and Device Design C-9 

Interfacing to a Microcomputer • • , C-10 



\ 



ERIC 



EV^U-UATION OF "FUNDANEiVT.US OF HIO^ RESOLUTION OF LITTOGRAPHY'', 
A Universit>' of California Engineering Extension Course Offered 
February 23, 1979. 



Evaluators : 

Martha Nlaxvell 
Caroline Showers 



Faculty Coordinator for the Course: 
Andrew Neurether, Assoc. Prof% 
Electrical Engineering § Computer 
Science, Dept. U.C. BopMceley 



About Lithography 

Lithography ii> one of the seven technologies in the electronics field 
that contribute to the development and manufacture of calculators, computers, 
and other electronic devices. In this field, the word "lithography** means 
the technique of writing large amounts of infoimation in the form of electri-. 
cal circuits on a very small area, the proverbial pinhead. The limits of 
high-resolution lithography, rather than those of the other six technologies, 
constitute the limits on production of a new electronic device. Now IS years 
old, lithography has for 5 years set the limits to all the other technologies 
in this field. 

Company training in lithography enables a new employee to make a con- 
tribution to the field in about 6 months. Experienced workers tend to stay 
within device fabrication (the conglomeration of the seven technologies) 
throughout their career, but have considerable mobility across technologies. 
Those who move into managerial positions usually have char , > of at least two 
of the seven technologies. The only formal training in lithography that a 
university student receives is that gained in research for a master's or 
doctoral degree on a relevant topic. 

Description of the Course 

Fundamentals of High-Resolution Lithography was a one-day course in- 
tended for engineers who are present or potential users of new lithography 
systems. The course brochire described tJie four lectures as providing 1) a 
practical conceptual foundation for the field, 2) a detailed discussion of new 
developments, and 3) a survey of alternatives for VSLI lithography with re^ 
duced linewidth capability. The topics included 1) Deep-UV Lithography, 2) 
a User-Orientated Simulator for Projection Printing (SAMPLE); 3) E-Beam and 
X-Ray Lithography, and 4) Composite Processing for High-Resolution Litho- 
graphy. These topics were presented by the speakers in lectures (ranging 
from 75-105 minutes in length) with a few ndnutes for questions after each 
talk and a 15-minute general question discussion period at the end of the day. 
Participants were given 15-minute coffee breaks in the morning and afternoon 
and a one-hour liinch. A social hour v;as scheduled at the end of the day, but 
there were problems in getting it set up so most of the participants left 
after the discussion session. 

Two of the speakers were from major industrial research centers and two 
were university professors. 
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METODOLOGY 

This evaluation is based on the results of questionnaires adminis- 
tered to 90 students at the end of the course, follow-up interviews with 

20 students, and interviews with the faculty-member- in-charge before and 
after the course. 

Student Questionnaire 

Evaluation questionnaires were distributed to stL«dent5 inmediately 
before the panel discussion at the end of the day, and were collected at 
the conclusion of the program. One hundred and thirty- tv« people attended 
the course and 90 questionnaires wejg^ returned. The questionnaire included 

21 items, only two of which were op^-^nded (participants were asked to 
describe the topics they found most valuable, how they would use them, and 

^ to make one suggestion for mproving the course.) All other items involved 
checking the appropriate alternatives or giving one- word answers stating, 
for exainple, job title or degree level. 

Items on the" questionnaire can be roughly classified into four groups. 
Demographic items (1-6) investigate amount of education, nature of profes- 
sional duties, and experience in lithography. Another group of questions 
(7,8,9,11,14,16,21) focuses on the effectiveness of the course: 1) How well 
participants' goals were achieved; 2) How information gained will be applied 
in the job setting; and 3) How well the course compares with other short 
courses for professionals. A third area of investigation (Items 12,13,14, 
16) is company attitude toward state-of-the-art courses. Finally, partici- 
pants were requested to indicate their interest in taking future courses 
and participating in follow-up interviews (Items 17,19). (See Appendix A for 
the questions and sunmaries of the responses.) 

1. Demographic Characteristics of the Students 

About half of the respondents described themselves as a manager, super- 
visor, or director. Yet, on the average, they spent 71* of their time in 
technical duties. About an equal number of students had Ph.D. 's. Master's, 
and Bachelor's degrees with 62% holding graduate degrees. Sixty-nine percent 
of the respondents had received degrees in the last ten years. Most worked 
in research and development; about 20% were in production; and a few worked 
in planning or marketing. Almost half the class had less than one year's ex- 
perience in lithography (or were not employed in the field.) 

Overall, the group appears to consist of well-educated, successful 
workers involved in technical R5D activities, but without a great deal ol 
experience in lithography. 

2. Goals and Applications 

Seventy- seven percent of the students reported that they took the course 
to update their technical knowledge and most were satisfied that the course 
met this goal. (39% reported it did so "very well".) Fifty-seven j)ercent 
checked "knowledge of fundamentals" as their goal and were somewhat less satis- 
fied with the course results (only 20% stated that it satisfied this goal 'Very 
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well".) It uas surprising to, find that almost half of the respondents 
with more than 5 years in *he field cited knowledge of fundamentals as 
their goal (see Ap.k, p.li) Forty-eight percent of the group took the 
course to gain a perspective for decision-making and most were satisfied 
that the course . helped in this regard. Meeting others in the field 
and finding out what they're doing was a goal of 30% of the group, and, 
despite the short length of ttfe course, most of these reported they 
attained this goal "somewhat". 

When asked to predict ijimediate and Iqiig-term applications of the 
course topics, about one-third of the class stated that they found infor- 
mation that they 'jould use iJimediately in Deep- U-V, SAMPLE, or conpatible 
Processes. Over hylf of the group stated that the material on E-Beam and 
X-Ray would affect their work in the future. (Note: these processes have 
not yet been fully developed for most industrial applications.) A list of 
the topics students found useful and ways they plan to apply them is in- 
cluded in Appendix p. 6. 

Most respondents expected to report information about the course to 
their supervisors and/or colleagues. Since only 10% mentioned that their 
supervisor suggested they attend the course, it may be that these will be 
informal reports. Thirty percent of the students said that the course 
topic reflected a major research area in their companies. This figure may 
reflect the importance of the topics and the effectiveness of the course in 
meeting engineers' needs. Also, 30% said they planned to take further 
courses in the field and 391 expected to do research individually on topics 
of concern to them. 

These results suggest that individuals' goals were achieved reasonably 
well and that most students expect to make use of the material presented. 

3. Company Involvement 

One- third of the respondents mentioned that their companies or super- 
visors initiated their decisions to attend the course. A similar nimber 
mentioned that their companies strongly encourage attending such courses, 
or that attendance might increase their possibilities for promotion or a 
raise. Since all of the students' fees for the course were paid by their 
conpanies, industry is obviously amenable to engineers' participation in 
this type of short course. The fact that 70'» of the participants requested 
their companies' backing in taking the courss does suggest, however, that 
the taking of such courses is regarded as a matter of individual initiative 
rather than a deliberately encouraged company policy. 

•As mentioned above, participants expected to pass on material presented 
in the course to their cclleagues and supervisors. Some stated that they 
would encourage their companies to do further research on these topics. Thus, 
the topics appear to be important and relevant to company interests. 

In general, students in this course seemed to be a highly motivated, 
self- selected group of professionals. 
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Conments : The sample of students replying to this questionnaire may re- 
flect an age bias. Of the 90 students returning the questionnaire, 39 
said that they had less than 2 years' experience in lithography. 

Perhaps younger ind newer eiapW^es are more willing to take the 
time to complete a detailed qu^tionnakre than are older managers. It 
may be that recent employ««a-.are more kpt to attend courses, or that they 
interpreted the title, "Fundamentals of High -Re solution Lithography" to - 
indicate that the course covered more fundamental matters than it actually 
did. 

.Another sampling bias may be that the people who answer question*- 
naires may be more favorably disposed .toward the course than those who do 
not. 

4. Comparative .Evaluation of the Course 

Eighty percent of the respondents found the pace of the course about 
right. With one exception, those who found it too fast had less than 2 
years' experience in the field. Nbst of those who found it too slow had 
more than 5 years' experience. The course was rated jjositively in com- 
parison with other professional and comnercial short courses attended by 
participants. A majority of those replying described this course as 
superior to others. However, this question was answered by only 35 people 
while the nimber of responses for all other questions ranged from 67 to 
89. Since only \2% of the group had taken this course before, it may be 
that the majority of students had taken no other courses in lithography on 
which they could base a comparison. 

5. Student Suggestions for Improving Course 

Students were asked to give one suggestion for improving the course 
and about half of the respondents complied. Most frequently mentioned areas 
were improve course materials (17) , emphasize more basics (11) , and provide 
more practical information (6). (See Appendix A, page^for a list of these 
suggestions) . <^ 

Snimary 

This course attracted a fairly diverse group of engineers in regards 
to background and experience in lithpgraphy. About half the group had 
little or no experience in the field vhile others had spent many years in 
lithography (long enough to be considered pioneers.) More than half of the 
class had earned graduate degrees and most held responsible positions in 
RSD. The average student was fairly well satisfied with the course and 
found one or two ideas that they could apply to their work ijimediately. 

Interview with Faculty Coordinator 



The faculty coordinator was inte:v ..- ^d before he was given the re- 
sults of the course questionnaire. He s s ..'.sied to predict how the group 



SUMMARY OF STUDENT EVALUATIONS OF FWDAMgNTALS OF HIGH-RES O LUTION LITHOGRAPHY^ 
( February 23, 1979)* 



I. What Is your job title? 

15 Technical staff or technician 
3 Research staff 

Engineers: 

3 Unspecified 

1 Field E. 
8 Process 

4 D/5velopment 

2 Masking 

1 Project E. 

I Product E. 

1 Photoresist 

2 Associate E. 

1 Software Analyst 
1 Electrical E. 

Other: 

1 Specialist 

1 Vacuum Tech. 

1 LSI circuit design 

1 -^Layout designer 



1 Physicist 
I Chemist 

Managers: . 

3 Unspecified 
i QA engineering 
1 Toch. development 
1 -^ch. liaison 
1 Dlgi^^'l systems . 

1 Applied En9:in«^iring 

2 Mask making 
1 QC 

I FPROM development 

1 Design and prcress d^;:velopment 

2 Research director 

Genera; Manager, OA office 



1 EBS engineering 
1 IC process engineering 
1 Operations head « 
1 Microlithogri phy and process 
labs 

1 Microprocess photomasking 

1 Engineering manager 

1 MOS design 

1 CAD manager 

I Technical director 



S<^l.es engineer 
Salesman 

Vice-Pre.>ideut . Marketing director 



* Percentages represc^nt the fraction of participants responding to each question 
who gave e certain ^^aponse. For example, 33% Ph.D. means that 33% of the 
parcicipanr^ who responded to Item 2 gave Ph.D. as their highest degree. 



Highest academic degree?^ 

331: Ph.D. 

29X H.LJ M.A. 

377. B.S./B.A./A.A. 

Year received? (69Z of respondents had received degrees tn the last 
10 years. ) 

Frequency Distribution — 72 respondents 
I97d 77-75 74-72 71-69 68-66 65-63 62-60 59-57 41 
8 14 16 11 10 4 ' 5 3 1 

What percentage of your time Is spent In: 
managerial duties? ^ 
technical dutleb? 

37. Individuals (422 of respondents) were full-time technicians (that is 

spent 90Z or more of their time in technical duties.) 
2 individuals (2% of respondents) were full-time managers (spent lOX or 

less of their time in technical duties.) 

Respondents averaged 71Z of their time in technical duties, fkvee people were 
in marketing. 



Do you work primarily In: 
f ^ 



72 SOI R&D 

212 Production 
6 91 Capital Equipment planning or marketing 

How long have you been in lithography? 



27Z 
237. 
32X 
18X 



I year or less 
2-5 years 
more than 5 years 
not in lithography 



Frequency distribution — 79 respondents 
0* kl i=A Izi Zzi 9-10 11-12 13-14 
T4 25 * 7 • 10 6 9 2 2 



15-16 



,28 
1 



What were your goals in taking 
til is course: 



571 
30t 



knowledge of fundamentals 
updating technical information 
perspective for decision-making 
meet others in the field and find 
out what they're doing 



Number 
responded 

64 
32 
22 



How well vete they achieved? 



not at all 
52 



somewhat 

36 
63 

77 



very y e 

72 



52 
6Z 



39 
31 
23 



*Figures in this column represent responses 
checked betveen somewhat & very well. 



Breakdown of Goals By Number of Years in Lithography 



€•-1 P,2 



Knowledge of fundamentals 
Updating technical information 
Perspectives for decision-making 
Meet others in the field 



5 Years 
or Less 

60Z 

66Z 

43Z 

25Z 



More than 
5 Years 

48Z 

88Z 

56Z 

40Z 



8. How did you hear about this course? 

(Frequency di8tributioa~8S respondents) 



Brochure 



68 



News 
Release 



Prof. Society 
Newsletter 



Company 
Newsletter 



Colleague 
17 



Other 



9. How well did the pace of the course suit you? 

80Z About right 
13Z Fast or top fast. 
7Z Too slow 

Did you find anything in 
the course that you can 
/ use immediately ? If so, 

in what areas? 



Deep U-V 
Sample 

E*Beam and X*Ray 
Compatible Ptocess 



32Z 
38Z 
15Z 
35Z 



Did you find anything In 
the course that will affect 
your work in the long term 
If so, -^at? ' 

38Z 

36Z 

56Z 

36Z 



ERIC 



74Z of the respondents to this question mentioned at least one topic for 
immediate use; 79Z mentioned at least one topic for long term use. 
(See appendix lor li^st) 



10. Have you taken this course before? yes 12Z 



no 88Z 



11. How does this course compare with other engineering short-courses you have' * 
taken? (in usefulness, content, interest) 

This question got a poor response (40Z of all persons returning questionnaires). 
57Z of the respondents rated this course as superior to others they had taken; 
43Z found it about the same; no ne gative responses were given. 
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12. Who initiated your decision to attend this course? 

you IJX supervisor's suggestion 30X selected by company 3Z 

(Note: some students checked more than one alternative) 

13. How is your attendance supported/recognized by your company? 

100^ payment of fees and expenses 24X company strongly encourages attendance 

released time i 1^ attending courses increases possibility 

of promotions/raises 

14. Will you report the information gained in this course to your company 
coUegues? 83X your .supervisor? 75X 

s 

15. Do you intend to seek further instruction or do individual or company 
research on the topics in this course? j 

30% major company research area 
30X plan to take future coijfses 
39X will do research on my own 
- am not sure 

\ 

16. Are you interested in taking courses in: process-modeling e-beam 32t 
other , Students mentioned the following courses: 

Basic processing concepts ^ 
Overview of process 
Semi-conductor 
Critical measurements 

X-Ray, resist • , \ 

LSI design • 

Photolithography 

Sputtering 

Dry processing 

Deep U-V 

(842 of students returning questionnaires ansvered this question) 

17. What one suggestion do you have for improving this course? 

(Responses are coded by area. Numbers In parentheses are the number of persons 
giving each suggestion. 43 students 7-48Z of the respondtuts — answered this 
qu^tion. ) 

Facilities .(6) 

1 Non-smoking area 

1 Uncomfortable seating 

1 More food 

2 Bad lunch 
I Crowding 

Lecture notes (17) 10"^ 

Most commented that notes were out of order. Al/O # 
I More detailed diagrams 
1 Complete bibliography 

3 Suggested a pre-course handout 

1 Desired copies of all diagrams presented 
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C.l P. 4 

Suggestions for Improving Course (contM) 

Logistics (2) ^■ 

1' Better timing 

1 Have monitor speaker in A.M. 

Length (I) t 

1 Two-da/" course would be more practical 

More basics (11) 

Assume less advanced knowledge 

Present more basics/ a basic understanding/ basic principles 

Hore detail, basic theory 

Include basic optical theory of projesliion 

Offer an intermediate level course (this one was too advanced) 

More practical information (6) * . 

2 Less theory 

3 More practical material 

1 More production information 

Other (5) 

1 Atfd an introduction to each talk (an overview) and have a slower p^ce 

2 Cover less material 

1 Better organization of material . 
1 Improve presenters* styles 

\ 

18. The NSF Evaluation Project Team Is planning to conduct follow-up interviews 

with englne^s who have taken short-courses. If you are willing to participate 
in a brief " interview to be scheduled at your convenience, please fill in your 
name and phone number below: 

N « J£ prefer personal interview at my company 
N • 25 prefer telephone interview 
do not wish to participate 

I do 58Z do not 42X wish the information I givia to be released to my 
company. 

/ 
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D«8crlpClon8 of mosc valuable copies and how they will ^a used.' (26Z of che 
group answered this question) * . 



Code Numbers: 



5 

7 

10 
17 

26 



30 

31 

32 

35 

41 

44 

47 



Elegant method for monitoring coapletlca of resist development 
by use of double ejcporsures. Also, Interesting future 
possibilities with Sample— not ready for It at this time. \ 

I found the talk on E-Beam & X-Ray quite Inserestdng. I will 
srudy the handout In^formatloD In more detail and use this 
information in evaluating Incoming E-B«am-generated masks. 

Deep U-V: Photoresist development, endpolnt control 

^ample^lnoiedlAte use of 'prograa ' 

Deep UV and E-Beam &^ X-Ray-^proJect planning 

Sample nost likely will be utilized for device design and 
masking criteria. • 



Sample—to model project loc^ printer Images 

E-Beam & X*Ray~miftklng decision for optimum buy date 

Compatible process—clues of problems 



Deep UV— Introduction to tlvft topic » renewed Interest applicable 
to future work * 
"SS^MBple— appreciate' awareness of the information, but the topic 
is too large for such short presentation' 

E-Beam '& X-Rs/y — not inmediately applicable to my work. Useful 
information for next generation of devices. 
Compatible process— most interesting of the presentations. I 
would have enjoyed more in-depth presentation. 

Deep UV & E-Beam— VLSI size decrease 
Sample — Design for fine lines for VLSI 

Compatible process — current process design and development 

J> 

Radiation damage, lift«-off techniques, computer program 



E-Beam & X-Ray~ better uncerstanding 6t mask generation 
Compatible process--set up and evaluation of plasma etch prpcesses 

Sample — development determination technique, program for process 
model 

Compatible process— pattern transfer technique study 
E-Beam & IC-Ray — alignment strategy, architecture. 
Might use deep \P1 techniques, in processing 

s 

The information presented will be used as reference material 
for future R&D worl in the area of microwave semiconductor 
devices. 
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Most valuable Topics in Course (conc*d) 

50 E-B4am & ^-^Ray — possible approaches to improving some of our 

lithographic techniques 

52 E-Beam & X-Ray— ^positiVe' «|?sist 

^7 Sample — opMrnization of, projection printing a 

E-Beam & X-Riy-n^ some considerations in E-Beam, selection of 
equipment 

Compatible process — selection of etching processes 

\. • 

58 I found the talk on cooi^atibie processes to be.very useful* 

Will use some of the techniques in making 'out microwave IC' >i« 

61 Compatible process — effects on plasma processlqg 

66 Deep *UV — presented best data^d" some usable ideas 

, E-Beam and Compatible process talks rehashed data in ).iteraCure 
and were pretty , much a waste of time, ^ 

31 biscusslon. practical problems and relationship to theory 

I in optical imagery, photreslst and «tch (plasma) processing, 

83 References for continue^ reading ^ 

E-Beam & X-Ray — possible uses in mask making 
Compatible process — reactive^ sputtering looks Interesting 

90 Deep UV and E-Beam — applications toN proximity alignment 
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EV-.UUATION REPORT ON TUB COMPOSITE NWTERIALS COMPUTATION WORKSHOP: A 
PRACTICU GUIDE TO DESIQJ AND TESTING OFFERED SY UNIVERSITY EXTENSION, 
UNIVERSITY OF CALIFORNIA, BERKELEY, March 26-30, 1979. '(Course Co^ 
ordinator: Dr. Stephen W. Ts^i, U.S. Airforce Materials Lab., Wright- 
Patterson Air Force Ba»e, Ohio.) 



The Composite Nfeiterials Computation Workshop is Intended to provide users And 
producers of composites with the most current guide to solving problems in d«ign 
and testing. The workshop uses lectures and drills to relate advanced methods to 
everyday problem solving. Participants are expected to be totally immersed in 
this five-day workshop and attend informal 2-hour evening sessions as well as the 
daily presentations. Participants are urged to bring their own progranmable 
pocket calculators and be familiar with their own operation. Advanced methods 
in design and testing are given on charts or formulae so they can be rAdily 
programmed on calculators and pre-programmed master cards of equations are avail- 
able for reproduction. The workshop topics include stress-strain relations for 
unidirectional composites, calculating" stresses in laminated composites, changes 
due to fluid absorption, mathematical modeling of curing and swelling, fatigue, 
fundamentals of testing and evaluation, and latest methods in micromechanics . 
The course coordinator is assisted by three speakers and several lab. assistants. 



This -evaluation is based on the responses of 35 participants who returned 
follow-up questionnaires sent two months after the course ^ended axvi on phone 
interviews with four participants. Since only 54% of the 65 people who enrolled 
in the workshop- returned questionnaires, the responses analyzed may not be repre 
sentative of the total class and the results should be interpreted with caution. 

t 

Student Characteristics . ;The people who enrolled in this course were diverse in 
background. Practical experience with composites, and in the type of composite 
material their work involves. Nine held bachelor's degrees, 16 held master's 
degrees, 5 were Ph.D. 's and one had two years of college. Seventy-nine percent 
had worked in composite materials less than five, years, and 411 were new to the 
field with l^ss than one year's experience. Ntost worked in research and de- 
velopment and spent most of their time in technical duties. Their work involves 
a wide range of products from tires to teeth to reentry vehicles. In 60* of the 
cases, the participant enrolled in the course on his own Volition; 401' took the 
course at their supervisor^ suggestion or were selected by their company. 



Evaluators: J^rtha Maxwell, Ph.D. 

N.S.F. Project Director 



Carol in Showers, M.S. 
Research Assistant 




This project was funded by. the National Science Foundation under 
Grant " SED-78-22138. The opinions, finding, conclusions, ahd recom- 
mendations expribsed are those of the authors, and do not necessarily 
reflect those of the>Jational Science Foundation. 
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Evaluation RejxDrt on Composite Materials Computation Workshop 



Student Evaluations of^ the Course. On the average, students rated this course 
as better than others they had taken. 14t of the group rated it as the best 
they had taken. No one rated it less satisfactory than other courses. 26% had 
taken no other course in this field. 

Most participants took the course to improve their Vnovnedge of fundamentals 
and job skills, and to update their knowledge. A smaller percentage checked 
"meeting others" and gaining "perspectives for decision-making" as goals. Hiose 
whose goals were gaining a perspective for decision -making, updating infoimation, 
and meeting others were quite satisfied with the course, but thc^se who expected 
to learn fundamentals and to improve their job skills felt the course did not 
succeed a<- well. Thirty-four percent of the students said the course did not 
help them learn fundamentals at all. 

The majoritv' held that there was too little emphasis .on applications ( >7%), 
a higher percentaj: ) than is typical in the other engineering extension cour ses 
we have evaluated. 

Most students rated the instructors' emphasis on fundamental!? and on current 
research as "about right." However, between 14 and 20t of the group stated that 
there was too little emphasis on these areas. 

.The average respondent rated the pace, length, and difficulty of the course 
as '.bout right, however, lit felt the course was too short. Since the course 
lasted 5 days, including infoimal evening workshops, this suggests that students, 
once introduced to a subject through a brief extension course, may becomb in- 
terested in a more formal, semester- length course or it may show that some 
students were less prepared for the course and need more individualized assist- 
ance than others. 

The teaching methods used by the instructors were judged as "good" by the 
average student with 31° rat'ng them as "excellent." 

Half of the respondents indicate that they had used some of the techniques 
or ideas from the course in their work within two months after completing the 
course, Eiehty-two percent feel that what they gained from the course will be 
useful to Then in the future. The ideas/ techniques that they have been able to 
use and tlwse that they consider most valuable range widely across the workshop 
topics. . .some gained very specific information, others felt they developed a 
broader perspective. Thost topics they considered least useful dealt mostly 
with the more theoretical presentations. 

Eighty-nine percent of the respondents made suggestions for improving the 
course. These were about evenly divided between suggestions for making the 
course more applied (i.e., more time on testing methods, make sure all programs 
are covered and that application problems are given, etc.), materials Cpi^vide 
texts on micTomechanics, better handout materials, handouts on all viewgraphs, 
etc.), and organization and methods (too many subjects touched on in too short 
a time; stick to a generally accepted notation and standardize among instructors, 
breakdown the course into beginning and advanceld sections, etc.). 
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Evaluation Report on Composite Materials Computation Wbrkshop 



Siirinary and Recomnendations . In general, students appeared to be well satisfied 
with tl\is workshop and most found something of value to them. Ho-vever, t)iere 
were some inconsistencies in the participants' reactions. A higher than average 
percentage felt that there was not enough emphasis on applications- -a curious 
result, especially since the brochure describes the workshop as being "explicit- 
ly application oriented" and states that all participants will be able to solve 
nunerous problems ijimediately aftei the presentation of the topics. Furthermore 
students were offered informal evening workshop sessions where they could receive 
individual help in applying the lecture materials by working problems. Perhaps 
th^rochure led the students to expect too much from the course. 

Three other factors may have contributed to the problem. . .the large size of 
the class, the fact that students work with such a wide array of composite ma- 
terials, and the fact that a high percentage were newcomers to the field. Stu- 
dents complained that some of the speakers used only examples from the aerospace 
industry to which they could no: relate. In evaluating other CEE courses we 
have found that ivhen there is a high percentage of inexperienced participants, 
there are more complaints about instructors' emphasis on applications, particu- 
larly when academic speakers are on the program. It seems that the students who 
is inexperienced in the specialized field has trouble in transferring general 
information to his work and needs very dear illustrations that relate directly 
to his specific job. In this co: rse, participants regarded Tsai's presentations 
as clear, fundamental, and applicable, but many found some of the other speakers 
too theoretical, not explicit enough, and not preoared enough for their needs 
(i.e., they complained about handouts, lack of copies of viewgraphs, etc.) These 
speakers may have assumed that most of the students were more sophisticated and 
knowledgeable in the field than they were. 

Planning a workshop for students with such a wide range of backgrounds and 
interests is difficult indeed. Some suggestions the course coordinator might 
consider in planning future courses are; 



1. Encourage speakers to prepare more complete handouts in- 
cluding copies of the viewgraphs presented, and alert them 
that some of the participants have very little background 
in their topics. 

2. Consider scheduling some concurrent sessions where par- 
ticipants could choose between two sessions--i.e. , one more 
basic, peiliaDS a review of a previous presentation or dis- 
cussion of fundamentals, and one involving state-of-the 
art research. 

3. Encourage participants to meet other participants who are 
working with their particular composite materials, if 
possible, so that they can share ideas and work out ap- 
plications of the ideas and techniques presented in the 
talks to their special needs. Students' needs and interests 
could be identified at the start of the workshop or in ad- 
vance if appropriate questions were added to the enrollment 
form. Also encourage the lab. assistants to identify students 
with similar interests. Participants can learn much from each 
other. 
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Evaluation Report on Composite Materials Computation Workshop 



4. Reexamine tH** jourse description to determine whether it 
accurately describes what students will get in the course 
and be able to do with the information they learn. Per- 
haps it is a bit unrealistic to expect that 65 different 
students from almost as many different industries, half 
of whom liave minimal experience with composites (and their 
present jobs) , will be able to solve problems immediately 
and get indivicJualized help on their problems. Since the 
course is a popular one, perhaps specifying some additional 
pre-requisites would help. 
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would answer sonte of the questions and to explain some of the findings. 
A sunmary of this interview is given below. 

Q. Whiy was the course dalled Fundamentals of High-Resolution 
Lithography? 

A. It seems that students responded to "fundamentals" in the course 
title and on the questionnaire as referring to fundamentals of 
theory rather than as an introduction to lithography. On the job, 
a worker will gain experience in the practicrl, applied aspects 
of lithography, but may never be exposed to che physics theory 
that underlies the various processes (e.g., electron and photon 
interaction) . 

. Q. Would you comnent on vhf you feel students saw E-beam and X-ray as 
topics that have potential long-term as opposed to ijimediate use- 
fulness? 

A. Currently E-beam is used for mask-making on the "original" surface 
but there is a possibility that E-beam may be used in the produc- 
tion process (i.e., making large numbers of copies) in the future. 
X-ray may be ready for use in the production process in about two 
years . 

Q. To whom was the course aimed? 

A. I aimed the course toward a class that I expected to be about 40% 
new to the field. These people would be interested in a list of 
references and an introductory talk on the topics. I expected 
another 40% to be engineers practicing at the state of the art 
level. These would be looking for a perspective on possible prob- 
lem areas and for new information. TWenty-percent of the group 
wjuld have contributed to the state-of-the-art knowledge. . .that is, 
these are my research colleagues vrfw.may even have written the 
references cited in the .xtures. 

Q. Do you think that the student's formal education (i.e., nunber of 
degrees) contributed to their interest in and understanding of 
the course? 

A. No -- degrees do not matter much, since there are no academic 
courses in this field that engineers routinely study. 

Q. What were your goals for the course? 

A. 1) Knowledge of fundamentals by which I mean background in physics 
including the nature of electron and photon interactions. About 
35% of the class time was spent on fundamentals. 

2) State-of-the-art knowledge -- that is, current information on 
Deep UV and E-beam. About 35% of class time was spent on this. 

3) Perspectives -- how to choose among alternative technologies. 
About 20% oi class time was spent on this objective. 
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Q. How io you think that engineer?? taking the course might apply 
♦".jv; Information iimediately and in the long-run? 

Iinnediate Use Future Use 

A. Deep U-V tricks buying new equipment 

SAMPLE tool 

E-Beam § X-ray understanding of insight; possibilities 

fundamental theory, for production 
masking 

Compatible Process silicon gate tricK 

A 

Q. What were your expectations when you prepared materials for the course? 

A. The handouts were designed to provide the student with bibliographi- 
cal references, and with copies of the viewgraphs used in the 
presentations. They were not intended to take the place of each 
participant's notes or to provide a substitute for presence at the 
presentation. 

Q. Do you expect the students to do further research or" take other 
courses? 

A. I expect that most students will do some kind of individual research. 
Ntost likely they would'use the list of references, look up a pub- 
lication, and call the individual. Then they might propose that their 
company do research. This might consist of a 6-month experiment rep- 
licating the study they read in the reference, a project that could 
take 10-1001 of one engineer's time. I don't expect that many of the 
students will take further courses in this field since the only other 
relevant one is a similar course given by Stanford's Extension Program. 

Q. How would you rate this course in terms of basic to advanced; practical 
to theoretical? 

A. I'd rate the course midway on a scale from purely basic material to 
purely advanced material. I assime that the students had some 
practical experience and were aware of the practical problems of 
lithography (as opposed to the theoretical ones.) « 

I'd also rate the course midway between the purely practical and purely 
theoretical although I expect that industry might see it as emphasiz- 
ing theory. While 50% of the course material dealt with theory lead- 
ing to or deduced from experimental research, this research is extreme- 
ly practical in nature, based on previous empirical observations. 
This is a field in which theory contributes little to practice, and 
thus 5 to 10 times more research in lithography is carried on in in- 
dustry than is done in the University. Research is thus molded by 
economic constraints, and theoretical research is not encouraged. 
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Q. How did you go atK5ut planning the course? 

A. First, I started with my own ideas, shared them with my university 
colleagues, and then tried them out on people in industry foi wham 
I act as a consultant. Finally I considered which speakers might 
be available. Thus, my impressions from professionals in industry 
about their needs contributed to my planning the course. 

In selecting topics and emphasis, I asked; 1) What is of current 
Interest, especially what are the new developments? 2) How ran I 
make the course complete, by adding Deep UV as well as E-beam anff 
X-ray? 3) Which speakers were available, including people from the 
university so there could be university involvement? I selected one 
speaker from U.C. to cover the current- interest topics (DeepUV; 
compatible process); for coverage on E-beam; and to announce the UC 
computer program) . Since more research goes on outside the univer- 
sity, it was important to get good speakers from industry. 

Q. What instructions did you give speakers? 

A. Each was given the 3 goals for the course, a brochure from a previous 
course, and an idea of the audience (i.e., 50* new, 501 experienced). 
EP'ih was told to include everything they would say on a viewgraph so 
that all the information appears in writing. 

Q. What have you leameii about teaching this kind of course? 

A. How to organize y that is, aim for a smooth presentation, break up 
the talks by subject matter, and use coffee breaks as dividers. 
How to prepare materials -- everything must be in writing. 

Q. How is teaching an Extension course different from usual university 
teaching? 

A. Preparation time is an order of magnitude longer because everything 
must be organized and in writing. The pace is faster because 
students are getting a one-sh(Jt deal and there are no rhetorical 
pauses for questions. The large size of the class also requires 
extra preparation. 

Q. What special tvn>s of problems do you encounter in a course like 
this? 

A. The wide range of background of the students is a problem. 



Al?o how many questions to allow -- that is., allow clarification questions 
now, but postpone discussion questions :ntix later, (^estions may extend a 
talk for 20 minutes or more. Industrial speakers often feel that they can't 
talk about their work because it is a company secret. One must seek out open 
speakers and find researchers who are good speakers and vice versa. The class 
orpanization also has to be carefully monitored because 100 people have a 
tremendous amount of momentim so that a lunch that is 15 minutes late can be 
a disaster. 
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Comments : The Faculty Coordinator appears to have a good grasp of the 
range of knowledge and abilities of the students who took the course. 
Fkjwever, in choosing to plan the course at a middle level (between basic 
and advanced and between applied and theoretical) , he set up a situation 
that was frustrating to many of the participants who expected a more 
introductory class. 

Recognizing the difficulties of planning and teaching a specialized \ 
course for engineers with such diverse experience (from the novice to the N 
very experienced professional), the following recomnendations are suggested 
as ways to improve the effectiveness of the course and student satisfaction 
without altering the basic course content. 



1. Reconmendatio ns. Consider changing the title of the course to something 
other than "Fundamentals of HRL".. The most dissatisfied students complained 
that che course did not give them the fundamentals they expected while satis- 
fied students were pleased that the course gave them a good overview of the 
field today. These latter students were more experienced, had taken more 
courses in the i?ield and/or were working in areas peripheral to lithography 
and attended the course to improve their general knowledge. If "Overview of 
HRL" is not acceptable, why not just call it "HRL"? 



2. Coarse Description . The course brochir 
the instructors expect that students shoul 
the course and the level at which the cou>- 
or two in the brochure that students cou7 i 
would be very helpful for some students, 
ences before the course, but might be ' 
Although specifying the prerequi jv; 
courage newcomers to the field fru . ' .v::s 
numb»»r and intensity of the complain* • ' 
by th'i course title and descriptior . 



describe explicitly what 
order to benefit from 
ht. Listing a reference 
e faking the course 
■ ^ot read the refer- 
.'t -firm afterwards.) 
^ u_ ' Sably not dis- 

' . \ \d reduce the " 
'vi* '■jyy »ad been misled 



3. Course Material^. Another freqiKfr 



course mau- - '*als. Each and ever/ on-" 
months aitei aking the course said h . ^ 
useful to them. Only two of those ii la. • 
the instructor hoped -- lo locate refv^*en^ , 
womplained that although the bibliograp^^.y wa-. 
graphs should be movi complete, others foLr< 
when they tried :? use them to review severa' 

remember enough of the course to understand them. Nor were they able to 
share the notes effectively with others in their company. Even the authors 
of state-of-the art research books and papers discussed in the course and 
those who had taken this course before complained about the notes and sug- 
gested that the instructors provide "stand-alone" notes. Other suggestions 
were : 



• emed the 
' Viewed 3-4 
id be more 
..«ater\als as 
f 1 even they 

notes and view- 
useless that is, 
ater,, they couldn't 



1) The notes should be a more complete outline of the 
lectures with the key terms and concepts given. 

2) The view graphs should be identified more clearly 
with better labels and some explanatory text. 
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3) Pages should be nunberecl and the graph j and other aids 
should conform to the sequence of the presentations. 

4) Taping the talks would help those who needed review to 
understand and remember the mass of information covered 
in the course. 



If the goal of the course is to help participants update their knowl- 
edge and apply state-of-the-art concepts to their work, then providing good 
notes seems essential -- particularly if the participants are expected. to 
share the information with others (or even convince their supervisors that 
they learned something from the course.) 

Extending the Course (or adding more specialized courses .) Almost half 
the participants indicated they were interested in taking a course in 
process modeling and several suggested that the course be extended to two 
days (with the second day optional) for a process -modeling workshop. An ex- 
tended course might be tried experimentally to see whether participants are 
willing to attend a longer course. Others want more specialized courses on 
some of the topics covered in this course, (i.e., X-ray) Taking this course 
seemed to whet the students' appetites for learning and although they com- 
plained about some of its aspects, and felt that much of what was said was 
over their heads, most planned to take additional courses in the field. 

5. Longer Coffee Break s. One of the aspects of a short-course that is 
important to many participants is the opportunity to meet others in their 
field and discuss what they're doing, therefore increasing the coffee breaks 
by 5 minutes vrould enhance this opportunity and increase the value of the 
course for these students.* Also it would offer more opportunity for a few 
more participants to ask questions of the instructors individually. 

6- A Simple Needs Assessment . To aid the instructor in planning" the courses 
(as well as being a way to help the participants feel that their interests are 
being addressed), adding a short questionnaire to the class registration form 
in the brochure is an inexpensive and effective way to collect this informa- 
tion in advance of the course. Questions could be asked about previous courses 
taken, work experience in lithography, interests and expectations for the 
course, whether they knew of and/or had used SAMPLE, etc. 



It's probably difficult to ^et students back into a lecture in 15 
minutes any;*ay, so planning a slightly longer break in advance would 
ease the pressure on instructors whose talks otherwise might run over. 



Tapes could be sold to the participants. 
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APPENDIX 



( ] ( ] ( 1 [ 1 
Follow-up Questionnaire - Composite Materials Workshop 
N- 35 

We appreciate your taking the time to answer these questions. Your response will 
help us improve our courses. 



1. What is your present job title? 

■(See p. iv) 

2. Your highest academic degree? 



3. Year? 



41-45 
46-50 
51-55 
56-60 
61-65 
66-70 
71-75 
76-80 



1 
0 
3 
3 
3 
4 
6 
9 



3 yrs. college(l); BA/BS(9); M.S. (16); 
Ph.D(3) 

4. What % of your time is^spent in: 

Average 

[151] managerial duties 
[831] technical duties 
[ 2%] sales/marketing 

5. Do you work primarily in: 

% of class 
[83] R § D 
[11] Production 

[^] Planning and Decision-making 
(~] Sales/Marketing 

(6 ] Other (specify) (1 teaching; 1 student) 

6. How long have you worked in composite materials? 1 yr . 2-5 yrs . 6-10 11-20 21-30 



N- 14 
l« 41 



13 
38 



3 
9 



2 
6 



2 
6 
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What were your goals in 
taking this course. 


8. 


How well were 


they achieved? 


% ot 


total class 


not at all 


somewhat 




very well 


[74] 


updating technical information 




[4%] 


[521] 




[141] 


[89] 


knowledge of fundamentals 




[34] 


[52] 




[14] 


(31] 


perspective for decision-making 




[ 6] 


[61] 




[33] 


[57J 


meeting others in the field and 
tinding out what they're doing 




[-] 


[70] 




[30] 


[80] 


ijrproving job skills 




[7] 


[64] 




[29] 


Rate 


the course on the following: 




• 










12. pace [6%] too fast 


[94* 


about right 


I-]. 


too 


slow 




13. length [4] too long 


[85] 


about right 


[Ill 


too 


short 




14. difficulty [4] too hard 


[89] 


about right 


[7] 


too 


easy 
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15. Have you been able to use any of the teclmiques or ideas from the 
course in your work? If yes, what and how? 

Yes 511 

.Vo or not. yet - 41t 

No response - S% 

(see coin7.ants attached p. iv) 

16. Did you ieam anything from the course that will affect your work in 
the long term? [B2\] yes [12%] no If yes, what and how? 

6% not sure 

(see p. V fi vi) 

17. How would you rate the teaching methods used in this course? 
% [31J excellent [57] good [9] fair [3] poor 

18. What was the most valuable part of the course for you? 

(see p. vi) 

19. HTiat was the least useful part of the course? 

(see p. vii) 

20. How would you rate the instructors' emphasis on the following? - 

too much about right too little 

fundamentals [4%^ [81t] 

application [-] [43*1 [57*1 

current research [9*1 [25*1 [20*] 

21. How did this course compare with other engineering short courses you 
have taken? (in usefulness, content, interest) 



[141 the best I have taken 

(31) better than others I have taken 

[29] about the same as others 

[ 0] less satisfactory than others 

[26] have taken no others 

22. Who initiated your decision to attend this course? 

I = [60] you [34] your supervisor [6] selected by company 

23. What one suggestion- do you have for improving this course? 

31 students gave suggestions (see p. viii) 

117. 
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The NSF Evaluation Project Team is planning to conduct case-studies on engineers 
who have taken short -courses. If you are willing tOf participate in twc brief 
interviews, to be scheduled at your convenience, and to permit us to interview 
your colleagues and supervisor, please fill in your name and phone nimber below: 



Name 



Phone No. 

- (area code) 



( 1 prefer telephone interview 
[ j want further information 
[ j do not wish to participate 



I do [ ] do not [ 1 wish the information I give to be released to my company. 
( 1 Please send me a surmary of the final report. 

Please return this questionnaire to U.C. Extension-XT37, 2223 Fulton St., 
Berkeley, C\ 94720. 
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Q. 1. Vt'hat is your job title? 

Mechanical linginoor (2) . * 

Research Chemist 
^ Research Engineer 

Research Fellow, Tires 
Senior I-nginecr :(3) . 
' * Technical Group Leader 

Associate Dean of Engineering 
Research Assistant 

Project Coordinator CSr. Research Chemist) 
lingincer 

Member, Technical Staff 

Aerospace Tech. (Structural Materials Engineer) 
Supervisor, Equipment Q Test. Unit, Laboratories 
Research Analyst 
Design Engineer 

Re-entry Vehicle Design Engineer 
Research Pel low 
Consul tant 
Chemist 

Research Scientist 
Engineer HI 

Head, Office of Engineering Mechanics 

Advanced Development Engineer 

Task Leader, Ceramics and Glass . 

Manager, R5D 

Senior Project Engineer 

Q. 15. Ikwe you been able to use any of the tecliniques or ideas from the 
course? If yes, how? 

Stress analysis, calculation. 

Computer programs in predicting canposite properties. 

l>se knowledge of micromechanics and fatigue life of comixjsite to 
modify the polymer research programs in selection of polymers for 
testing and in design of experimental programs for composites life 
, prediction. 
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1 :iin bcy.ii\ning to use some of the programs which Wore presented. 
Also, sonic of the ideas for testinR with water ^orpt'iofi. 
Structural design for composite jxirts. 

Calculator programt' for determining laminate properties. 

Basic understanding of strain calcurat;ions due to moisture effects. 

lUfferent design of a component on helicopter rotor blade. Gain 
understanding of rationale of composite design of wing of aircraft. 

We are currently in a design phase where a composite material is 
pro|X)scd for a structural component. I needed this background to 
proceed with' the design. « 

• • 

Concepts to stress to others that composites are not isotropic materials. 

General atttitude- -design of test samples; • 

I had ^becn away from aerospace composite materials for 11 yrs. prior 
to last year. 1 have spent 20% of my time the last year advising cur 
plaster lab on applications with more structural advantage. 'The course 
reinforces and ujxiatcd my knowledge. I can now better understand com- 
ix)site material docunqnts I read and broaden the plastic lab application 
knowledge. . • ' 



It). Hid you learn anything in the course that will affcx:t yourVork in the 
long term? 

1 will le transferrctl to the RfiD of comiX)site materials. The course 
will be useful for this. 

llic theory will help in the design of chcmcial service equipment. 

Slxjuld help in our approach to composite design. 

Strain and stress transformations and composite technology. 

Reference materials; familiarity with T-59 calculator. 

Basic imcfcrstanding of viscoelastic effect and fatigue crack propagation 
beneficial in current and future research. 

In design for aircraft comix)nents in comjxjsitcs; select materials and 
com|)osite .structural design.- 

I met each of the Instructors, know Iww to contact thtm, and may ask 
them questions in the future. I was greatly impressed by all of them. ' 

Tlie Tsai-lNli failure criteria in design of aircraft comp^j^^e ' 
structures. ^^^^ 

Moisture, creep discussions will allow us not to worry' about these 
effects as much 

t20 
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• / I need considerable study time to 'apply what I've learned^ 
Sijnplc computational techniques using tl-59 calculator. 
Strengthened fundamental understanding. 

Methods for analysis of composite laminates* to improve efficient of 
composite structures. 

• ' . . \ ' '• • 

Waitinji for the go-ahead on some new programs. 

Teaching and .'research activities will be initiated this Slinner. 

Lack of understanding of molecular origins of failure of composite 
materials. Will put more research efforts in this area. 

TI-59 methods. 

Q. 18. iHiat was the most valuable part of the course to you? 
Mechanical properties. . ' 

Comjxiter programs. 

Fundamentals (3) * * •* 

Dr. Tsai's part (4). The manuals and the calculator programs for 
»^ calculating laminate properties. 

Learning that we wore already on the right track. 

Dr. Springer's |)art. 

Calculator programs, moisture absorption, micromechanics in that order. 
^ Laminate properties programs. 

Failure theory and application of the generalized ikx)ke's Law. 
Practical applications of last afternoon. 
Limited infm.'on fire hazard with electrical equipment. 
Lxchange of infm. with others in the field. 
Dr. Hahn's.- ' ~ 

Tlic broad -scope concepts and stat^-of-thc-art ideas. 
Vispoelastic effect, fatigue, fracture. 
Getting a perspective from knowledgeable people. 

Data and reference indicates composites still must be used with extreme 
care. 
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Practical aspects. 

(***i;vcr\' topic seemed to appeal to someone. The list above shows 
a great deal of variety of interests. However, here and in the 
interviews, the material covered by Tsai was much more popular than 
that presented by the other two speakers. Tsai's talk was focused 
more on fundamentals.) 

I . Q. 19. What was the least useful part of the course? 

Infm. on hydrothermal properties (Hahn) 

I|png Cqns. 

Part covered by Hahn. (3) 
, . Parts relating to aircraft applications. 

Micromechanics and theoretical models (2) (iialin again) 

« 

Needed to use TI-59 programs on sample problems in class time.. 
Time spent on detailed use of TI59 calculator. 
Static property calculations. ^ 
Any concepts not amenable to e/perimental verifications. 
Description of Boeing activities. 
I racturc mechanics was rushed, not given enough time. 
Afialvsis of fatigue. (3) 

Long theoretical devcloiments that resulted in few conclusion. 
Re|X)rts ol incqnclusive test results. 
Less ivory tower, more practical --work. some problems. 
J^iunihatc analysis. ^ 
Tlieoretical studies and derivatioI)f^i. 

l:ver>thing else besides. Tsai's part, because they did not reduce if 
to the simple form that he did, they did not have simple manuals like 
he did, and they gave their lectures on papers rather than teaching 
tlie material. 

Q. 23. Uliat one suggestion do you have for improving the course? 
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^torc Application 

t 

--(live niore application data in the commercial sense. 

--Less anphasis on theory, more on application (5) 

--Nbre emphasis on practicAl applications 5 testing- -there was 

almost no time devoted to testing methods. 
--3c sure that all programs are covered and that application 

problans are given. 
--Add material on manufacturing processes, limitations, ind the 

shop's effect on design. 
--Spend minimum of 1 day, 2 is better, just working sample problems. 

Material s 

--Have all lecturer? prepare manuals like Tsai's. Teach rather than 
iwrticipate. 

--Provide texts on micromechanics, environmental effects, and fracture 
mechanics and time -dependent properties analysis. Also improve 
lecture techniques, particularly the micromechanics part. 

--Suggest review of mechanics of materials and matrix methods before 
taking course. Have complete texts available. 

--Better handoiit materials. (3) 

--All viewgraphs should be available as handouts (2). 
--Prefer to have materials before attending course. 

. Organization and Instructional Methods 

--Too many subjects were touched on in too short a time. Cut a few 
out . 

--^bre thought should be given to the introduction given before each 
lecture on what the speaker is trying to show or prove. 

--Stick to generally accepted notations and standardize among 
instructors. 

--A step-by-step design of some component would be helpful. 
--Add section on latest developments in composite materials. .eg. , any 
' new materials being investigated now, why, and how. 

--It sliould be more concise. 

--Break down course into two areas --advanced and basic topics-given 
entirely scf^arately. 

Information from Interviews. 

I tow luive vou used in format ion/ ideas from the course? 

--Used in a NASA project to develop and fabricate ar windmill plate. 
Hie course gave knowledge of how to calculate laminated stress and 
calculation programs to calculate surface life at high pressures. 
I hn^'en't changed procedures but have written proposal for changes 
and submitted it to my supervisor. The use of calculators will save 
us dollars by replacing use of full -sized computers. Tsai's pre- 
sentation was really good.. the best. Hahn and Springer's work is 
only of academic interest- -I need more practical information. During 
the last day of the course, invite an engineer to present state of 
the art of composites in industry. The main benefit of the course 
was that I was introduced to how to do calculations of stress-strain 
irt composites. I never learned this in school. 



123 



t 

c' 2 P. 17 



-Course jiave me a good working knowledge of composites, l-xcellent 
information for future use, but our dept. hasn't much use for this. 
I would like to take further courses extending this with more 
applications- -eg. buckling. 

-Calculation of poisson's ratio using Tsai's notes. Tried to share 
with colleagues but this is tough unless the group had taken 
course. .they're too used to homogeneous and isotropic materials. 
Tsai's presentation excellent but I didn't get as much from other 
people. They were more directed toward aerospace and got into 
design of wing for aircraft--too speci^c. Fifty percent of the . 
course was of interest to me. Clarcmoiv^ Hotel was a good facility; 
beautiful view. Course could be improved if slides on actual tests 
were used and there were more discussion on significance of testing 
and its pitfalls. Need more emphasis on test than special applica- 
tions. I'm not sure who course was aimed at. Wouldn't help to have 
notes in advance; ixK)ple would just follow notes, not presentation. 
Half were not prepared anyway.' Tsai's presentations were dynamic 
and handled well. Dr. Mahn was very well qualified but couldn't get 
|K)ints across in front of a group. He puts people to sleep by talk- 
ing in a monotone. I talked to him on the side a couple of times. 
Tsai should take up 50% of program. 
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Information in this report is based on questionnaires completed by 17 
participants in the April 9-13, 1979 class, 8 follow-up questionnaires 
completed by participants in the November 1978 class, interviews with 
8 participants, and an interview with Robert Mitchell, the course 
coordinator. 



The material in this report is based on work partially funded by the 
National Science Foundation under Grant. No. SED- 78-22138. Any 
opinions, findings, and conclusions or reconmendations, expressed in 
this report are those of the author and do not necessarily reflect the 
views of N.S.F. 
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What is VE? 

Value engineering is a procedure to reassess the economic vi- 
ability of a project previously planned or designed. If the project is 
currently in its initial design stage » value engineering aims at saving 
money. As an example, consider the construction of a concrete channel 
for a river bed or aqueduct. Conventionally, one builds a frame for the 
sides and pours concrete to form a vertical slab. A value engineering 
approach might find it more economical to pour a horizontal slab and lift 
it into the vertical position. 

There is a certain methodology/approach to design/way of thinking 
which is called value engineering and which consists of a specific 
series of questions which one follows in assessing one's project. This 
is what is taught in the week- long workshop. 

Description of the Course 

The Value Engineered Design and Construction course is a 40-hr. 
workshop organized and taught by Mr. Robert H. Mitchell, one of 175 
Certified Value Specialists (CVS's) in the United States. The Society of 
American Value Engineers (SAVE) of which Mr. Mitchell is a Regional Vice- 
President, certifies CVS's and approves the training workshops that thfey 
teach. While the Value Engineering (VE) methodology is fairly standard, 
any course outline approved by SAVE ma/ be used in the 40-hour traiTiing 
session. The training may be geared to a specific discipline this one 
focuses on design and construction applications-- or to more general en- 
gineering problems. Participants who complete the workshop are certified 
as eligible ro biccme members of a Value Engineering team (described be- 
low) . There ?iie many additional requirement for CVS certification includ- 
ing the stipulation i\7t fifty percent of one's working hours be spent in 
performing value engin«ser\ag analyses. Thus most participants in this 
course will continue to gain lixperience in '/E by working on VE teams, but 
will not go on to receive CVS certification. 

The client of a design or construction firm may require that a VE 
study be done in addition to conventional design procedures. The VE 
analysis is a separate Ixidgetary item and may be subcontracted to a VE 
consulting firm. The team leader for the VE study must be a CVS so if the 
study is not subcontracted, the firm usually has to hire a CVS leader. 

The leader may require the remaining four to six lAembers of the team 
to complete a 40-hour VE training program. Members are usually selected 
by discipline (that is, mechanical engineer, electrical engineer, architect). 
Sometimes the client requests that the team exclude anyone who has previ- 
ously worked on the design project. 

Private fiims that operate on a profit- loss basis have typically been 
most interested in the cost- saving and efficiency results of VE. However, 
government agencies are showing increased interest in VE, and some agencies 
require that a VE analysis be done on larger cor^struction grants. (Note: 
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The federal govermient's irtcreased concern with VE may be reflected in 
a difference between the November '78 and April '79 classes-- only one 
participant in the Nfovember class identified himself as working for the 
government while 47% of the April class reported that they worked for a 
govemnental agency.) 

Course Goals . The goals of this course are twofold: 1) to train 
engineers in value engineering and 2) to enable conijanies to save money. 
The payback to the client of a VE study is usually much greater than the 
cost of the study and of training the team members. 

According to the instnictor, students who take the course benefit 
by gaining an increased potential for satisfying clients, a chance to 
establish personally a rtew profit-earning center within their canpanies 
and increase their success, prestige, and opportunities for promotions. 
The company gains similar benefits: new profit centers, new ways of 
thinking for employees so that they do a better job, higher work satis- 
faction among employees, and more clients. 

Methods and Instructors 

Mitchell uses the Learn, See, and Do method of instruction-- tha^ 
is, lectures, films, and teamwork. Students learn by doing-- that is, 
through, participating in a workshop project that they may brir^g jji, them- 
selves or a "real-life" project that is assigned by the insttlictors. 
Team study is emphasized and teams are organized on the first day of the 
class to build student interest. Also, Mitchell intersperses lectures 
with team sessions so that students can apply the ideas they have just 
heard and absorb the new material better. 

Mitchell divides the VE system (and the course) into five phases: 
1) information, 2) speculation, 3) analysis, 4) de^^elopment and 5) pre-, 
sentation and follow-on. The class participants role-play different 
specialists (architect, contractor, construction engineer, etc.) as the 
teams work through each phase. Mitchell uses only Certified Value Spe- 
cialists as instructors ana tries to be personally involved in at least 
two workshops per year to observe the quality of the instruction. He 
often lends course materials to other CVS instructors. 

Students and Their Problems . A week prior to the start of the 
course, students are sent the text and a list o£«^ked publications. 
Mitchell statjes that they tend to arrive at the c6urse somev^t emotion- 
ally upset and apprehensive because value engineering methods call into 
question conventional design procedures. Most students are practicing 
engineers or other professionals, but a few are supervisors. Most higher 
level employees, that is, managers or executives, attend .four • -to eight- 
hour VE seminars that introduce them to the concept of VE but do not cer- 
tify them to become a member of a team. 



Some students come on their own initiative; others are preparing 
to work on a VE team for v^ichthey have already been chosen. 

Mr. Mitchell stages that the course may cause problems for students 
because it requires them to change their fundamental beliefs. In per- 
forming a >E study,, conventional designs may be questioned or discarded 
and clients' criterk and requirements for a project may be challenged. 
Students learn that twenty percent of one's design efforts account for 
801 of the cost. This means that they must" continually be open to new 
knowledge and must accept the idea that they must always be reeducating 
themselves. Also, they learn to ask questions, both within the work- 
shop and on their jobs, even when it means <loubting the conclusions of 
.their supervisors. 

Methods o 

Questionnaires were administered to the 17 students took the 
April '79 course during the last day of class. Also 8 out of the 11 
participants in the November 1978 class returned follow-up questionnaires 
mailed to them five months after the course ended. .Seven students (6 
from the '79 group and one from tJie '78 group) were interviewed by phone 
three to sLx months after they had completed the course. 

RESULTS 

Student Evaluations of the Course 

Students , gave the VE course positive ratings in conparison with other 
engineering short courses they had taken. Only one respondent said that 
he felt that it was less satis^'.ctory than other courses. The '79 class 
was somewhat more positive about the course than were students in the '78 
group. 

Q. How did this course conpare with other engineering short courses you . 
have taken? (in usefulness, content, interest) 



'78 


•79 




131 


lU 


the best I have taken 


38 


53 


better than others I have taken 


38 


24 


about the same as others 


13 


0 


less satisfactory than others 


0 


18 


have taken no others 



Ntost of the students from each class cited "learning methods for 
economic evalv.at .ons" and "reducing design and construction costs" as 
their goals i.. taking the course and indicated that they were "very well" 
achieved. Some checked "receiving a certificate of participation" and 
some checked "finding specific design alternatives"-- a goal the group 
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felt was only "somewhat" achieved. Responses to this latter item seemed 
to be influenced by whether the student's project had been selected. for 
team-discussion. That is, 8 of the participants in the '79 class brought . 
projects, but only 4 were used in the workshop. Those people whose pro- 
jects v«re chosen felt the course served them very well; those whose pro- 
jects were not selected were less satisfied^ 

Q. What were your goals in How well were they achieved? 

taking this course? 



# 


responses 


not at all 


somewhat 


very well 


121 


121 


•78 


•79 


'78 


•79 


•78 '79 


4 


10 finding specific design 
alternatives 


0 • 


20% 


loot 


sot 


0 301 


6 


15 learning methods for 
economic evaluations 


<> 

0 


0 


28 


20 


79 80 




16 reducing design or con- 
struction costs 


• 14 


0 


43 


31 


57 69 


4 


13 getting a certificate 
of participation 


20 


IS 


40 


15 


20 70 



Participants gave high ratings to both the team workshops and lectures, 
although more students in the ^79 class judged the lectures to be excellent 
than did those in the '78 class. On the average, both groups rated the films 
as "good" although several students conplained that they were out-of-date. 

Q. How effective were the various forms of instruction? 





poor 


fair 


good 


excellent 




'78 '79 


•78 


•79 


'78 


'79 


•78 


'79 


team workshops 




12% 


181 


38t 


291 


SOI 


531 


fi 1ms 




12 


18 


63 


59 


-25 


24 


lectures 




12.5 


6 


63 


44 


12. S 


50 



The IP'^S class rated the instructors' en^hasis on fundamentals, appli- 
cation, theory, and current research as about right although several students 
fell that there v/as too little emphasis placed on current research. 
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Q. How would you rate the instructors' enphasis on the following?* 



too much about right too little 
'78 '78 '78 ' 

fundamentals 12 t 881 0 

application 0 88 12 

theory 0 100 0^ 

current research 0 63 37 * 



*This question was not included in the questionnaire given to the 
'79 clzwss. 

Sunnary . The students surveyed in these VE courses were more satis- 
fied with tKe course and gave it higher ratings than did students in the 
9 other CEE courses we evaluated. Even those who felt the course content 
was "basic cannon sense" considered the class experience was worthwhile 
(with one exception.) 



The Satisfied vs The Dissatisfied Customer-- Examples 

Data from interviews with two of the students with similar backgrounds, 
but diametrically opposed attitudes about the course, provide interesting 
insights into some of the factors affecting students' acceptance of the 
course. Both students completed their bachelor's degree almost 30 years ago; 
one designs steam plants, the other designs sewage disposal plants. Neither 
are government employees. Both stated that they had always used VE princi- 
les informally in their work, although they were not familiar with VE termi- 
nology and methods until they took the course. The satisfied student's 
project was selected for the team workshop; the dissatisfied student did not 
bring a project, however, other factors may account for their attitudes to- 
ward the course. The dissatisfied student did concede that the course could 
be beneficial for others, but not for him. 

Case A . 

A structural engineer with 28 years experience in construction rated 
the course as the best he has taken. Although he had not previously studied 
VE, he has taken a great many UC, CEE courses on other topics. He felt that 
the VE course met his goals of learning methods for economic evaluation, 
and reducing constn»ction costs very well and was sc3mewhat satisfied in learn 
ing how to find de'^ign alternatives. He described the instructors as excel- 
lent, interesting, and very knowledgeable, thought the lectures and team 
approach were excellent, and the films good. 

He said that he was familiar with the principles underlying VE and had 
always used VE informally-- though not according to this method, but felt 



the course was very worthwhile. As a result of taking the course ..he has 
changed his approach tp design-- becoming more analytic, using more brain- 
storming, and considering more options in a design. His project was dis- 
cussed in the workshop and he feels he profited from this .. although his. " 
recomnendations were too late to be incorporated in the design of thef 
plant. 

He liked the ''ength of the course and its focus on real-world prob- 
lems especially since "you got involved in the problem and stuck with it 
for a week." 

He felt the course gave him a tool to use plus a reference, based on 
an accepted system, to support the conclusions he would make in terms of 
any project. He added that he felt the course would benefit managers as 
well as the practicing engineers who took it. His one criticism was that 
he thought the instructors should stress the difference between the points 
that are to be used as givens with little or no change vs. those that are 
suggested, but where variations are optional. 



Case B . 

The student who was most negative about the course rated it as less 
satisfactory than others he had taken, felt his goal of reducing design 
or construction costs was met not at all, and reconnended that his company 
not send other engineers to this course. A senior engineer with 29 years 
oT'experience, he said that he had no specific knowledge of VE methods be- 
fore taking the course, but found that he anH the others in Ms company 
^were routinely using VE methods in their work. In designing a project, 
he and his colleagues ^eavch for cheaper ways to do things according to 
functior. So he felt no need to change as a result of taking the course. 
He does not need k VE certificate and is not workinj; on projects that re- 
quire VE analysis per se. He did concede that th** course vioyld be of 
great benefit to practicing engineers who did not stress tliese ideas in 
their work and suggested that it would be good to give a sijnilar course 
for process designers and show them how VE can affect " process in which 
you are working with a given piece of equipment rathe; than designing the 
1 ize of a window. He felt that the course instructors ranged from ex- 
cellent to good and knew what they were talking about in terms of content. 
He rated the team workshops as good, and other aspects of the cuurse as 
fair. 

> 

Expected Applications and Inpact of Course on Job 

The miJiority of the students in each class had applied or planned to 
apply VE principles to their job duties and half of the '78 ijroup had en- 
couraged their colleagues to take training and their companies to 5ipply VE 
to all of their projects. 
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'78 


'79 


151 


2i% 


50 


24 


50 


41 

• 


63 


88 



As a result of taking this coursci I (plan to) or (have): 



taken additional training in V.E. . 

encouraged my colleagues to take VE training 

.'^couraged* by company to apply VE to all of 
their projects 

applied VE principles to my job duties 

Of the five workshop phases-- information, speculation, analysis, 
development, and presentation, most students checked analysis as the one 
that was most ijimediately useful. However, the majority of the students 
' in both classes indica'ted that they could use all of the phases inmediately 
(or had used ^em) . Developsnent and presentation were most often cited as 
being useful in the future. IVo persons saw the phases only as useful to 
them in the fCfture. 

Which topics (will be) or (have been) most useful to you in your work 
eit*^er immediately or in the long-tenn? 

* 

Of ijirrediate use Will use in future 





'78 


'79 


'78 


'79 


Infoimation 


loot 


761 


43 


35 


Speculation 


86 

• 


56 


45 


29 


Analysis 


iOO 


88 


43 


24 


Development 


57 


71 


86 


41 


Pre^ntation 


51 


59 


86 


53 



(Briefly described the ideas and techniques you found most valuable and how 
you used them.) 

The seven engineers \rfio were interviewed three months after complet- 
ing the course were using VE principles in a variety of projects. Even 
those who had not been able to convince their companies to adopt VE formal- 
ly said that the ideas gained from the course helped them in their own work 
(indeed some said it changed their lives.) (See Appendix for sixnmaries of 
these responses.) 

Suggestions for Improving the Course 

Ten students gave suggestions for improving the course: 
projects (getting more information about how 

projects were selected and what to bring in, etc.) -- 
4 responses. 
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films (need updating) 2 responses 

more emphasis on economic analysis (life cycle costing, etc.) 

2 responses. 
other (see Appendix)- 2 responses. 

Company Support for the Course 

Students in the '78 class were more often chosen by their Supervisors 
or companies to attend the course than were those in the '79 class where 
the majority came on their own initiative. 

Q. Who initiated your decision to attend this course? 

'78 '79 

I did 13* 53; 

Supervisor 49 35 

Selected by Co. 38 24 

(some students checked more than ofie alternative) 

Companies paid full or partial course expenses for all the students 
surveyed. Although the course coordinator indicated that VE training would 
improve possibilities for promotion, only one person checked this alter- 
native. Perhaps studencs are not yet aware of this potential^ or are re- 
luctant to check this response (i.e., it may suggest unseemly! ambition.) 

Q. How does your company supj^ort/recognize attendance at courses like 
this one? 



'76 


'79 


2S% 




88 


94% 


50 


24 


0 


6 


0 


0 




6 



company strongly encourages attendance 

ccnpany (^.ys fees and expenses 

company grants released time (time off) 

course attendance increases possibility of 
prcjT^o t ion -raises/ 

no particular encouragement 

other (specify) Co. pays part of costs. 

(some students checked more than one alternative) 



None of the participants said they were required to have VE certifi- 
cation to hold their present positions, but four anticipated that VE cer- 
tification Would be required 5oon. 
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Educational Background 

Most of the participants had bachelors degrees with only 3 holding 
master's degrees. Most completed their formal education more than 20 
years ago. 



Q. What is your highest academic degree? 



Year graduated? 





iNov. 


•78 


April 


•79 




Nov. 


•78 


April 


•79 






X 


' N 


JL 




N 


% 




% 












1948- S5 


2 


40 


8 


50 


H.S. 






3 


18 


1956-60 


2 


40 


2 


13 


A.A. 








« 


1961-65 






0" 




BA/BS 


5 


21 


13 


76 


1966-70 






5 


31 


MA/MS 


2 


29 


1 


14 


1971-75 


1 




1 


6 



Work Experience 

Students in the '78 class more often identified their primary work as 
design while those in the '79 class worked primarily in engineering or con- 
struction management. 

Q. Do you work primarily in: 



Nov. 


•78 


Apr. 


•79 


N_ 


1 


N 


% 


6 


75 


6 


38 


1 


12.5 


7 


44 






4 


25 


1 


12. S 


5 


31 



Design 
Engineering 

Construction Management 
Other 



Nbst of the students in both classes had worked more than 10 years in 
their present specialties. 

Q. How long have you worked your present specialty? 





Nov. 


•78 


April 


•79 




N_ 


1 


N_ 


% 


1 yr. 


1 


14 


S 


29 


2-4 yrs. 


1 


14 


2 


13 


5-10 


• 2 


29 


S 


29 


11-24 


2 


29 


0 


0 


25+ 


1 


14 


S 


29 
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On the average., particpants /Spend about 25% df their time in managerial 
duties, 57% in technical and, 8*» in market ing/kpntrac ting. Two are full- 
time nianagers (spending more' than 901 of • their^;p$x-in administrative 
dutie's and 4 are full-time technicians.) Most oTthe group, as befitting 
their experiences, supervise athei/s (ranging from 1-10 employees.) _ 

Conclusions ■ ^ 

This course was the most successful of the 9 CEE courses we have 
evaluated in both satisfying students and in. influencing their behavior 
on the job. Those [who took the course were very positive -- indeed 
^enthusiastic-- abdit the format, teaching techniques, quality of the in- 
struction, and the applicability of the ideas they learned. Fbllow-up 
data suggests that even when they were not able to lise the VE methods 
formally, they were applying the ideas to their ovn design projects and 
daily work. Students who took the course expressed greater confidence in 
their ability to solve design problems-- a feeling of efficacy (that is, 
a stronger belief in their own professional capability.) 

What is surprising about these results is that the students were 
practicing engineers with many years of experience. They were undoubtedly 
veterans of many CEE courses and most had not taken graduate work so- that 
one might expect them to be a bit jaundiced about taking CEE courses. 

The interaction of a number of factors seems to account for the 
success of the course. First of all, the students were a relatively 
homogeneous group (in comparison to most CEE courses) in regard to edu- 
cational background, interest, and experience. Second the small classes, 
team workshops, real- life projects, and the course content itself maximized 
the conditions for transferring the ideas learned. Third, the instructors 
were viewed as both knowledgeable and effective. 

Recommenda t ions 

1. Several students complained that they needed more explicit 
instructions about how to prepai^ projects so they would be 
selected for the team sessions and the criteria used in 
selecting projects. They expressed disappointment that their 
projects were not chosen and felt that had they known what was 
expected they would have brought more background infoTmati(m 
to the class. However, in our opinion, the description and 
criteria presented in the April VE Course brochure are very 
explicit and clear. It may be thjit prospective studt^nts do 
not always read the course brochure, and we are not sure how 
that might be remedied. (Or maybe they read an old brochure.) 
Perhaps sending out another copy of the project information 
with the materials sent to the students in advance of the course 
might help, if this is not already being done. 

2. If it is essential to use vintage films in the course, then the 
instructors should consider ways to explain this so students 

-> will find them more palatable. Students reject films that are 
dated and may miss their messages. 
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. Some participants will inevitably be disappointed because 
they seek simple answers to complex questions and find that 
the instructors do not (anc* perhaps cannot) tell them preci- 
sely which alternative is best or how to oetermine the best 
•among an array of possible alternatives. Perhaps the instruc- 
tors could emphasize the importance of the negotiation that 
inevitably goes on when group decisions are made. It would 
help students if they would point out and discuss the struggles 
that occur during team decision-making. 

Other than these small points, .ve recommend that the in- 
structors keep on doing what they are doing. Tt seems to be 
working very well indeed. ' 
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APPENDIX 



Q. Describe the ideas and techniques you found most valuable and how you 
■plan to use them: 

(April 1979 Class-Questionnaire) 

(10 responses) 
I-unctional Analysis (4) * 

-- Functional analysis; use in design and construction; possible 
environmental . 

• I 

Functional analysis of elements. These techniques will be used 
on highway construction projects and resource conservation 
studies. 

Found the idea of functional analysis.'most valilable. 

-- The identification of function followed by examination of alter- 
native solutions in an organized manner. I am^^ojm to encourage 
staff members to use this approach in planning ijewvtor^ ' 

Overall Course (3) 

-- This short "40 hours" course is most valuable. An excellent 
approacli to the following: 1) the today inflation and 2) very 
educational approach in human relations, not only for profes- 
sionals but in general daily life. The melody of a song makes 
a person happy, so a jgood approach will be awarded. The late 
>^Ghandi once said: "One' friendly word can carry fruits tomorrow." 

-- The entire program of value management will be utilized on our 
in-housc projects, both design and construction. Anticipate 
doing VE work for present and future clients. 

Formalized methodology for the gathering and analysis of in- 
fonration for both original design and review of existing 
construction. '. 



Other As|>ccts of the Course (3) 

-- The tcclviiquc of eliminating the apparently imix)rtant criteria 
and zeroing in on the main REQUIRED criteria is the most 
valuable idea. 

-- S}x?culation--I will try several different ways to do the j^b, 
and yet perform the basic functi6n. 
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The most helpful is the process. Also, the course was a great 
motivation to me to spend more time on the process. I would 
like to work on a VI: team to get some practice. 

(Follow-up Questionnaire Nov. '78 class) 



Techniques used. 

- comparing alternative structural systems, building 
skin materials using a matrix for relative ranking. 

- identifying and concentrating on primary function to 
avoid getting sidetracked by peripheral or irrelevant 
issues. 

-- Reviewing a project at early stages to determine cost and syst«n 
alternatives. 

(Follow-up Interview Responses) 

Mow have>ou used the ideas/ information presented in this course? 

I have been reviewing sewage treatment plant design and my project 
was discussed in the workshop. I made reconmendations that were 
too late to be incoriwrated in the design. I discussed the course 
with director and assistant and gave them the course notes. In 
my own work, I am analytic in my approach, use more brainstorming- 
and consider more options iji design. 

-- I've used it on a small project in an informal way- -a small retro- 
fit for air conditioning in two equipment rooms in a hospital. ' 
Asked did they need air conditioning, etc; what sort of down time, 
etc. (anticipated complete retrofit of system with some down time; 
instead they suggested adding a unit without losing service time). 
I've issued an office memo about the course and VE techniques and 
nK?ntioned it in a committee meeting. In my personal work, I 
changed to using terminology and method in dealing with clients 
and colleagues (asking what the building is supposed to do.) I've 
used the notes to check on questions raised and approaches, and to 
review phases of the VE method. 

-- I hope to use VF soon in work for A.C.F., possibly for the Air 
Force Base. Have shared information with colleagues although some 
of them have taken the course before and have membership in the 
Society of American Value Fjigincers. I ve not clianged procedure's 
because most of the staff apply VE subconsciously in designing 
structures. Have used the notes once or twice to keep acquainted 
with the material. Dc lot plan to use procedures per se, but will 
edit them. 
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Tve used the ideas on a project at Vandenburg Air Force 
Base for a space shuttle- -considered the electrical aspects. 
I shared ideas with colleagues and supervisor very briefly, 
and told them tliat it was a very good course, one of the best 
I fiave taken. Before taking the course I would just use a 
standard method to select a switchgear; now I look at the 
function and recoinnended a less costly switch gear that per- 
formed the same function. Also there were. things that were 
not ne«ded in a design- -for instance, a lightning protection 
system (there's no lightning hazard in Cal.) 

-- Having taken the course, may awareness has changed. I'm more 
conscious of $'s. My office is working on a hospital design 
project for which we will be doing a VE analysis soon. I'm 
pushing colleagues to use VE in the hospital project and recom- 
mended that they take the course. Since I work for the govern- 
ment, any project over $2 million must have a VE analysis done, 
so there is no choice. The individual can't change the process. 
Have used the course notes to get an overview of the process 
when I was pushing for the VE analysis of the hospital project. 

Tlie project studied in the course has been submitted for approval - - 
a roadside rest project. Figure we've saved $10-15000 oh a $2-3 
million dollar project. Have made scMne smaller reconmendations 
on road construction projects resulting in $10-12,000 savings from 
changing shape of culverts, slope of adjacent land, etc. 

I've no control over whether VE analysis is done in company. The 
VT. tcajn coordinator chooses the projects for VE analysis. I have 
discussed the course, the project studied, and conclusions with 
my supervisor. Have used the course notes to refresh my memory, 
and to look up procedures vrfien attending meetings on the roadside 
rest project. 

-- Yes I've used them- -qualified by the fact that I have always used 
V'l: in designing power plants. Part of design is to look for 
cheaper ways to do th^fgs and to design according to function. 

Q. »vhat one suggestion do you have for ijnproving this course? 

(10 responses) ^^P^^^ ^^^^ Class-Questionnaire) 

Projects (4) 

-- Better review of projects. 

-- More advance information as to what is expected in the course, 
i.e., needed much more reference material to properly evaluate 
design of project. Would have brought it if I had known. 

-- Better selection of projects. 

Need to bring more information into the classroom with us. 
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Films and Materials (2) 

-- Uixlate films ASAP. 

Update films and samples. Inprove quality of printed matter. 
More emplmsis on economic analysis (2) 

More emphasis on ijnportance and, procedures of economic analysis. 
-- Ntore explanation ^hd instruction on life cycle costing. 
Other (2) 

The team workshop--! think that more tim^ is required to 
investigate and analyze the project being VE. 

# 

-- Stressing those points that are to be used as given with little 
or no change, and those that are suggested ways but variations 
are optional. . 

Q. How much did you know about the topics covered in the course before 
you took it? 

-- quite familiar--28'years in construction. I've always used VE 
informally, but not according to this methodology. 

-- very minimal except what I'd heard about the course itself. 

Architects do VK naturally. . .consider how you choose material 
for building. I can't see much difference between common sense 
and VH. * . . 

I was not familiar with the approach to VE problems but was aware 
that other ways to- approach a problem exist in order to cut costs, 
but not aware of the approaches. 

-- I was familiar with the ideas from a previous course as I had 
taken a 40-hr in-house workshop. 

* • 

-- I liad read some literature and the reading suggested for this 
course. Prior to taking the course I had been assigned to a VE 
team. • . ' • 

Had no specific knowledge of the VE method. 

Q. How do you feel about your knowledge of VE now? 
I feel the course was very worthwhile. 
-- Peel my knowledge has increased considerably. 

Have not had much opportunity to practice, hope to use it soon. 

Uo 
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-- Heel i know 851 of what there is to know about VE. 

-- TfK? course was wortliwhile reinforcing information learned 

in other courses, but the specific approaclies were a little • 
clifferent--i.e. , the Information Phase had not been included 
in previous course. 

Yhe course was worthwhile although I haven't felt a lot of 
results. It's hard to change designs and get changes approved 
by the original designer. 

-- No change in knowledge as I was alr-^ady using the ideas presented 
in the course. 

Q. Itow do you feel about the instructors' presentations? 

Very competent; could have put more enphasis on mechanics of. 
economic analysis or suggested where to learn this information- - 
perhaps from another course. 

Rxcellent instructors comparcu to those in other courses I've 
taken. 

-- Excel lent --very understandable. 

Did their job adequately as r?»onitors for this course. What you 
get out of the course depends on the activity of the team you're 
assigned to. Mine had top-notch, enthusiastic people so it was 
excellent. But instructors did their job adequately. 

"1 ^'^^ good. This was one of the better courses I*ve attended. 
Both of the instructors were very enthusiastic which made the 
course go over very well. 

-- Mitchell --excellent; Kelly-good (in some cases he appeared to be 
talking from rote). Content: Both knew what they were talking 
about. 

Q. Was the content of the course focused on the needs of practicing 
engineers? If not, how might it be improved? 

-- Yes, largely because of the workshop approach. Instructors should 
bring in projects and assign them to groups. Some people were too 
involved with own project because they had designed it to be able 
to criticize it adequately. My group was divers; and unfamiliar 
witii the proj<;ct so this was good. 

-- It's not possible to do this in a 40-hour course. Should be 

offered differently for people with different levels of experience. 

Yes, it was appropriate. I would like a longer study period to 
Kork on an actual job. Workshops were good, but we needed more 
time to study the problems and more discussion of finances --how 
to make a cost statement. --More options than I expected; need 
explanations on when to use each. 
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Yes --but instructors could have monitored the workshop p,roup 
activity more actively. Tlicre were disagreements ».Uhin the 
jjroup that instnfctors might have helped to arbitrate and 
resolve. 

Yes. It's hard to change your concept into VE, to consider 
function analysis rather than cost analysis. Need practice-- 
5 days in a classroom is not enough to make that switch, but 
can think of no better way to learn/ teach the VE method. 

-- Yes, depeiiding on the type of engineer. If the engineer did 
not normally stress these ideas in his work, course would be 
of great benefit. 

Q. KTiat were the main benefits of the course to you? 

-- J'irst, I obtained the VE certificate; my office now can accept 
government projects. Second, I've learned a very good design 
approach which I've spread to other workers in my office. 

-- To serve clients who think there is value in the service and 
to serve the ppblic (taxpayers) money especially in an infla- 
tion economy. (We do government projects.) 

-- Opened my eyes to these concepts. 1 always took the standard 
methods for granted. Now I question them and use .* functional 
approach in design. An excellent course, I was very surprised. 

-- I got an appreciation of VE and the principles which carry over 
into fields outside my own. It is a general problem-solving 
methodology that can be useo in everyday life. 

-- The course got me thinking in a different direction. Wliat does 
a design do rather than v^at does it cost? 

-- Don't feci I got any benefit from the course because I was 
using the principles routinely. The VE certificate was hot 
needed as I am not working on a VE team or on projects that 
reciuire VE analysis per se. 

-- (bvc mo a tool to use plus a reference to support the conclusions 
I would make in terms of any project. Course is good for the 
management level as well as the engineers who took the course. 
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EVALUATION OF 1TIE aECmON-BEAM LITOGRAPHY COURSE 
OFFERED BY CONTINaJLNG EDUCATION ENGINEERING 
1>.C. EXTENSION AND THE COLLEGE OF ENGINEERING 
UNIVERSITY OF CALIFORNIA, BERKELEY 

November 27, 1978 

(Professor T. E. Everhart, Faculty Meniber in Charge) 



Evaluators: Martha Maxwell, Ph.D., 
N.S.F. Project Director 

Carol in Showers, M.S. 
Research Assistant 



Electron- B^am Lithography is an intensive, one-day, state-of-the-art course, 
with the purpose of .presenting both the advantages and disadvantages of direct 
electron -beam fabrication, with enough fundamental background material that 
attendees will be able to decide for themselves whether direct electron-beam 
micro fabrication should pily a role in their future production plans for high 
performance integrated electronic circuits. The faculty -mJsmber in charge, 
Thomas E. Everhart of U.C. Berkeley was assisted by three speakers, Alec^ N. 
Broers of IBM Research Laboratory, Hans Pfeiffer of the Electron Beam Technology* 
. Group of laM, and Larry F. Thompson of Bell Laboratories. The topics covered 
in the course included fundamental aspects of electron-beam lithography, direct 
electron-beam exposure' systems, electron-beam resist and processing consider- 
ations and opportunities in nanometer fabrication. A panel discussion was held 
at the end of the day where all speakers were a\'ailable to answer questions from 
' '» the audience. 

This report is based on the responses of 54 course participants to follow- 
up questionnaires sent six months after the course ended and on phone interviews 
with seven participants , Sin:e only 50 percent of the participants in the class 
returned questionnaires, the responses received may not be representative of the 
total class. Therefore the results should be interpreted with caution. 

Student Characteristics : The participants in this short-course ranged widely 
in educational background, experience in the field, positions held, and interests 
There was a fairly even division among those holding Bachelor's, Master's, and 
Doctoral degrees. They received their last degree over a 25-year span with the 
. .majority graduating after 1965. This class seems less involved in technical 
duties and more concerned with administration than is t>T3ical of participants in 
other CEE courses. The average respondent spends about a third of his time in 
managerial duties, 4*1 in technical^ and 5% in marketing. Over half of the re- 
spondents spend more than 20 percent of their time in administrative work while 
a third devote more than 901 of their time to technical work. Most participants 
work in research and development with a small percentage in planning and market- 
ing (12%) and production (81). Respondents indicated their experience in lithog- 
raphy ranged from none (20t) to more than 5 years' (29%). 

This project was funded by the National Science Foundation under 
Grant "* SED-78-22138. The opinions, finding, conclusions, and recom- ^ 
mendations expressed are those of the authors, and do not necessarily 
reflect those of the National Science Foundation. 
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Student Evaluations 

Most people enrolled in the course to update their technical infwmation 
and the majority reported that the course was successful in this regard. Those 
who wanted to learn fundamentals or acquire perspectives for dec is ion -making 
were somewhat less satisfied with the course. Generally, participants felt the 
pace of the course was about right and only three people complained that it was 
too slow. The average respondent rated the instructors' emphases on fundamen- 
tals, applications, theory, and research as "about right." About a quarter felt 
there was too little emphasis on applications, 181 wanted more theory, and 9% 
would have liked more research. Overall, students were satisfied with the course 
ranking it as somewhat better than other CEE courses they had taken. Tv» said 
it was the best course they had taken, and five rated it less satisfactory than 
other courses.* Twenty- two percent said they had taken no previous courses in 
lithography. 

Forty percent of the respondents found something in the course that they 
could use imnediately on their jobs, and a slightly higher percentage felt 
information learned about topics in the course would affect their work in the 
future. About a quarter of the respondents reported that they were using ideas 
or information from the course in their current company projects while 401 had 
done additional individual research. TVenty- three 'percent said they had done 
nothing further with the information.* Ten percent of the group had enrolled in 
other lithography courses while the majority had discussed the course content 
with their colleagues and 431 had discussed it with their supervisors. Most re- 
ported that the course materials were useful, although a few felt them to be of 
no value. 

Most respondents felt thecowrse had benefited them professionally and that 
their companies had gained "somewhat" from their attendance. All but three of 
the group said they would recommend the course to a colleague. 

Twenty- four people made suggestions as to how the course might be improved. 
The most frequent suggestion (n»12) was that the course be made longer and/or 
that topics be covered in more depth (i.e., more fundamentals, more application, 
more time for informal discussion, etc.) Six people madi suggestions about the 
course materials (better materials, send h idouts out in advance, etc.) A few 
conmented on instructors, organization, and other topics. 

The following sunmaries of some of the interviews- illustrate how different 
participants view the course. 
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.An analysis of the 5 dissatisfied students showed that all agreed that 
there was not enough emphasis on applications. Thiree had taken the course 
before and found nothing new. TVto wanted more depth. Two found the notes 
worthless and one felt that IBM dominated the course. 
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Intfciview Sunmarie s (E-Beam Course) 



1 An engineer manager in process development who supervises 7 people, has an 
M.S. and 8 years experience in the field. 

He felt quite familiar with the topics before taking the C0L-^se,,but had no 
hands-on experience with the equipment. After the course he felt more con- 
fident as he now has source material for understanding the different types 
of machines and their uses. N.y 

Prior to the E-Beam course he had attended a Kodak Seminar and has gone to 
the SPIE Conference since taking the E-Beam course. 

He presently is involved in a planning effort of an affiliated group in making 
, a decision io purchase an E-beam machine. He felt the most relevant part of 
the course was on process- types of resist, and has been helping* the group by 
contributing information he learned in the course on this topic. 

He has used the course notes/materials for reference £nd for deciding to 
purchase a machine. 

felt the E-geam course instructor's presentations were very well done, 
but assuned some prior knowledge. (He is not a user^in the field and would 
have liked and needed a presentation on fundamentals). He felt all three 
speakers were well -prepared andjj-ive nor A presentations. 

He reconwended that the course might be improved with a 1-2 hr. presentation 
on fundamentals and felt that Everhart's overview lecture was oriented to- 
ward state-of-art rather than an introduction to E-beam. (Commented that 
Ever hart was an excellent speaker.) He suggested that an additional day 
would have been gor J with the first day spent on fundamentals and the second 
on state-of-the arc information. 

He felt that the main benefit of the course was that it exposed hijn to the 
• state-of-art in both machines and processing techniques but he considered 
the discussion of capacity for E-beam was somewhat esoteric since these po- 
ter^tials are' not yet developed. 

In general, he was reasonably satisfied and felt he had learned a lot from 
the course. 



2 A company president who spends half his time on technical duties, supervises 
7 people and who received a B.S. some 25 years ago. 

His company is working on developing their oWn E-beam resist amoebus unit (a 
production machine) and he took the course to update his technical information 
(finding it somewhat satisfactory for this) and to increase his knowledge of 
fundamentals (and was well satisfied in this.) 
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Prior to taking the course, he had practical knowledge and as a result of 
the course he feels much better about his background --he learned of the 
state of the art and met other people and learned how they are thinking. 

He's taken no other courses in this ^ield and attended this one because 
it was held at the airport which is close to his company. 

\ He has used the infonnation from the course ito look up patent^MU)ther 

X work, to get in toucR with researchers in testing and R§D, and to set up 

\ replication experiments.* His interest is the chemical development of 

photo resist ^Kd is limited to building the instrument itself. He hasn't 
changed procedures and is still in the process of checking other people's 
results. He gave the course materials to those in his company who will 
direct the project for new ideas in chemicals and for replication of 
experiments. 

He felt the instructors were all good. The Hughes and Bell people spoke 
Qn topics most relevant to his work. He felt that the others were well- 
qualified and interesting although he did not pay attention to their 
presentations per se, he did not fall asleep either. . 

He recommends that the course be spread over a two day period with more 
on the phemical and^ application techniques of resist. He felt there was 
too much on the equipment (design, and idiosyncracy) . He's just inC^rested 
in the product and doesn't need to know how the machine makes its product. 

The main benefit of the course to him was that it exposed him to information 
.on chemicals that have been tested, evaluated, and developed for use in 
photo-resist . 



*3 A NITS with a M.S. degree who has been in lithography 2 years. He said that 
his work is primarily technical and he uses the E-beam process to fabricate 
devices. Prior to taking the course he was not familiar with optics nor how 
• the E-beam machine works and said the course gave him a broader picture of 
the basic background of E-beam research. He has taken no other courses in 
this field. After the course he tried to apply the idea of developing re- 
sistor profiles as a function of the energy of the electron beam. The re- 
sults matched those described in the lectures but were not precise enough 
for what he needed. He says he has reviewed the notes from the course, sent 
some memos about it to his colleagues and given them the course notes. He 
has not had much chance to use the ideas from the course as yet. He thought 
the instructors were very good but felt they should go deeper into optics, 
E-beam resist, and fabrication- -for him the material was too basic. He liked 
Everhart's presentation best as he's taken courses from him. Considers TSomas 
and excellent speaker who gets the audience's attention. 

He feels the course was probably intended to present more background infor- 
mation than to give ideas for immediate application and suggests that a two- 
day course would be more adequate- -the first day could stress background and 
the second present detail. 
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He felt the course benefited him because it broadened the E-beam lithog- 
raphy picture and gave him the fundamental p\ sics necessary to understand 
it. 



#4. The president of an investment company. He took the course because he was 
interested in the future of tvo companies and wanted to learn whether thiey 
were keeping up with the state-of-the-art. A former philosophy major in 
college, he said that 2/3 of the course was over his head, but that he was 
well^ satisfied that he got what he wanted out of it. Inmediately after, the ' 
course he met with one of the other participants and was tutored- -got his 
questions answered and the language and concepts clarified. He observed 
that one of the companies that he was interested in sent five participants 
while the other only sent one so he called the president of the bpard of 
the latter company and arranged a meeting. He and the board president then 
met With the company president and the director of the lithography section 
and quizzed the director. He said that as a result he was satisfied that 
this company was keeping up with the field even though they only sent one 
representative to the course. He felt that the cc •'•se helped him to learn 
the vocabulary and the right questions to ask and clearw up a lot of the 
misconceptions he had about the field. 



S .'^ Ph.D. with university teaching experience who is the lead person in E-beam 
lithography in his company, but who has only had 6 months experience. When 
he took the course he knew nothing about E-beam, but was very well prepared 
in physics and chemistry, and feels that the course increased his knowledge, 
and familiarized him with the field. He said the course was very valuable 
as an introductory course, but was not state-of-the-art as it was too low- 
level, and did not delve deeply enougE into topics. He felt the instructors 
presented the material very well and suggested no improvements. He felt the '! 
course prepared him Very well for surmer conferences and that the timing of 
the course was good. The reprints handed out were helpful as they directed 
him to reference material. 



Recownenda t ions 

This course appears to be about as good as it can be considering the con- 
straints of time (one day) and the various needs of those who attend it. 
Participants rank the course as better than other CEE courses they have taken 
and, in general, are satisfied that it meets their needs in updating their 
knowledge. However, there appear to be some strong reasons for trying longer, 
perhaps two -day courses that would meet some needs that are not presently being 
addressed. A longer course would benefit those participants who need more in- 
tensive explanations of fundamentals and help in applying the concepts to their 
specialized situations, if an extra day of workshpp/discussion sessions were 
offered. Those people who have taken the course before or those who are experts 
in the field might be better satisfied if they had more opportunities to ex- 
change views with other participants and with the instructors. If the course 
were longer, sessions of particular interest to the more experienced lithography 
engineers could be included. 
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[I [1 [ 1 [ 1 (1-4) 

E-Beam Lithography Course Questionnaire 
^ (54 Qviestionnaires returned out of 108 sent.). 

Please fill- out the following items carefully. Most questions can be checked,, Kit 
a fe)/ require short answers. 



V 



e appreciate your taking the time to answer these question^. The informaticin you 
give wi^l be held confidential unless you agree to release it. 

(5) I do [161] do not [84t] wish the information I give to be released to ray 
compan>:. N-32 

(6) What is your job title? (7) Your highest degree? (8-9) year? N 

BA/BS-39t NVk/M5-26l 53-55 - T 

Wat J of your time is spent^in: ^^'^'^^ \ \ 

avg. i34j managerial duties (10-11) ...70 . in / 

[61] technical duties (12-13) 7? 7^ . 

[ 5] sales/marketing (14-15) v ^ ^ ^ J 

Do you work primarily in:, 
M CO [731] R 6 D (16) [71] Sales/Marketing (19) ; 

*^ [8 ] Production (17) . [0 ] Other (specify) (20) 

[12 ] Planning and Decision-making (18) 

How long have yd^ been in Lithography? 

0 -10 1 yr - 7 2-5 - 17 6-10 - 8 11-15 - 5 16-20 - 1 (21-22) 

What were your goals in How well were they achieved? 
taking this course? 

» responses Not at all somewhat very well 

(23) [46] updating technical information [TT] I4i»j [56tJ - 

(24) [33] knowledge of fundamentals < * [ -] [64 ] [36 ] 

(25) [29] perspective for dec is ion -making [10] [45 j [45 ] 

(26) [23] meet others in the field and 

find out what they're doinj (10] '[71 ] [29^5^ 

(2?) How well did the pace of the course suit you? 

(6tl too slow [94] about right [-] too fast 

N- 51 

Did you find anything in the Did you find anything in the 

course that you could use course that will affect youii . 

jjimediately?' If so, what? work in the long term? 

^ N < N 

E-Beam . U (28) "7? (29) 

Fundamentals 

Direct Exposure 12 (30) 18 (31), 

Systems ^ 1 4 ' ' • 

Resists Processing 19 (32)' ^^"^ 19 (33) 

Nanometer Fabrication 4 (34) _ 14 (35) 

(On tHe reverse side, briefly describe the ideas and techniques you found most valuable 
aiid how xpu have used them.) 







too much 




* 




(1) 


fundamentals N » 




[lot] 


(36) 




application N ■ 


51 


[ - 1 


(37) 




theory N « 


49 


[21 


(38) 




current research 


N - 43 


'[ 7 ] 


• (39) 
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How would you rate the instructors* emphasis on the following? * 

about right too little 

(3) (2) 

[78t] [12%] 
[73 ] [27^] 
* [80 ] [18 ] 

[84 ] ^ [ 9 ] • 

(40) How did this course compare with other engineering short courses you have 
taken? (in usefulness, content, interest) 

(4i] the best I have taken (4) 

(31] better than others I have taken (3) 

[33] about the same as others (2) 

[10] less satisfactory than others (1) 

[22] have taken no others (0) 

(41) Who initiated your decision to attend this course? 

[851] you [6] your supervisor [9] selected by company 

N - 46 

(42) How does your company support/ recognize attendance at courses like this one? 

[17^] company strongly encourages attendance (4) 
N=59 ^ company pays fees arid expenses (3) 

[16 j company grants released time (time off) (2) 

( - j course attendance increases possibility of promotion/ raises (1) 

[ - 1 no particular encoutagement (0) 

(43) Have you sought further instruction or done further individual or company ♦ 
research on the topics in this course? 

[261] major company research area (3) 

[41 j have done considerable research on my own time (2) 

[10 j have takgi additional courses (1) 

[23 "1 no (0) ■ ^ 

(44-45) Have you discussed the information gained in this^course with your company 
colleagues? [62%] *^upervisor? [43] 

(46) How useful have the lecture notes and/or other courses materials been to you? 
[6%] very valuable [80%] quite useful [14%] not useful 

G47) Do you think that taking this course has helped you professionally? 
[76*«] yes [13] no [11] am not sure 

N - 53 
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Evaluation of the Electron -Beam Lithography Course, 



(48) If yes, explain, 



(49) How much did your company benefit from sending you to this course? 

(19»] a great deal [70%] somewhat [lU] not at all 



Explain: 



N = 53 



loo 
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(50) Would you recoiraiend this course to a colleague? [94%] yes [6%] no 

N=52 

(51) What one suggestion do you have for improving this crurse? 



The NSF Evaluation Project Team is planning to conduct case-studies on engineers 
who have taken short-courses. If you are willing to participate in two brief 
intei^views, to be scheduled at your convenience, and to permit us to interview 
your colleagues and supervisor, please fill in your name and phone number below: 

Name 

Phone No. 

(area code) 

N 

[lO] prefer telephone interview 
[ 5] want further information 
[21] do not wish to participate 

[is] TM like a summary of the final report 

Please return questionnaire to: 

Engr. Extension »XT37 

U.C.B., 2223 Fulton, Berkeley, CA 94720 

Answers to What is your job title? 
Staff 

6 Technical staff 

Engineering science specialist 
5 Engineer 

Consul tinj^ member, engineering staff 
2 Project engineer 

Design engineer 

Process engineer 

Chief engineer 

Senior engineer 

Senior staff scientist 

Associate engineer 
2 Research associate 

Group leader 

Managers 

2 Manager 
Project manager 
Section head 
Photomask manager 
Technical liaison 

3 Laboratory manager 
3 Engineering section manager 

Systems operations 
Operations 
Physical science 
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Director 

Research 

Marketing 

V.?., Engineering 

V.P., Marketing director 

President 



Professor 
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Q. 27 Valuably Ideas and Techniques 

23) As this was my introduction to the fields the immediate result was 

to acquire the technical language and concepts for communication with 
others in the field. As far ^s long term, only, or so much, as a 
broader background always influences. 

24) Learned some techniques of value in designing eilecton-optical 
systems, especially from Hans Pfeiffer. 

38) Ideas on columns and processing, that I was able to bring back to my 
group and pass on to them for their follow-up. 

41) Purchase of e*beam system. 

49) Most valuable thing for me was the talk by Dr. Pfeiffer describing 
his IBM direct writing machine. The talk was very detailed, reveal- 
ing most of its **secrets.** I work for a company manufacturing e-beam 
lithography ecquipment. The practical details are useful for me. 
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Q. 48. How DiJ the Course Help You Professionally? 

« 

I'm new to semiconductor industry. This provided good background. 
Useful in managing new pro jct. 

Gives me insight as to direction towards which researchers in other 
companies are applying their money and time. 

Now know capabilities of e-beam Jithography, 

It was a good in-depth introduction to e-beam lithography. 

Better perspective on e-beam systems and techniques. 

General knowledge, trends. 

It has provided a good background of material related to future 
decisions. 

It has given me general background concerning c-beam processing. 

We will be forced to make a decision w.r.t. e-beam lithography. I 
think we're better prepared. 

Useful for my decision making, as a general e-beam technical informa- 
tion and knowledge for me. 

My job is to develop the competing/complementary technology of x-ray 
lithography so I must have a good understanding of e-beam lithography 
concepts and procedures. 

The main value was in discussing the material covered with the 
instructors (after hours). 

I now know what IBM has done--at le^t, the order of magnitude, I 
also know how poorly the market was analyzed in 1975 and how tightly 
IBM can hold secrets. 

Gained better overall impression of state of the art in e-beam 
lithography--what 's commercially available and its limitations 
compared with IBM/BELL/TI systems. 

Currently making decisions related to e-beam technology within the 
company. 

Tliere will come a time when I will be using e-beam equipment. Courses 
like this will make NBK more effective when the time comes to purchase 
equipment. 

Knowledge of the state of lithographic art helps me evaluate plans 
for our research activities. 

Had little general knowledge of lithography at the time I took the 
course . 

Awareness of customers* fabrication techniques and general knowledge. 

IC design and fabrication is the most important future activity for 
engineers in our company. 

Personal contact and stimulation of ideas. 

Maintain technical understanding. 

I gained a much better understanding where the state of the art is-- 
which c^s an industry watcher is very important. 
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P- 48. How Did the Course Help You Professionally? (Cont'd) 
Provided background. 

Update of information is clearly useful. 
Professional contacts, useful information. 
Wider scftpe of knowledge in this area. 
Industry contacts were helpful. 

> 

In my company we do not have any immediate plans to get involved in 
exotic technologies like e-beam. We do however need to keep abreast 
' of the major developments in the field. I found the course to be 

useful for this purpose as I have found ethers of the Berkeley courses 
that I have attended. 

Information obtained has added to my ability to make appropriate 
decisions regarding this subject. 

Q. 49. How Much Did Your Company Benefit ? 
I don't know. " 



Not At All: 

Course benefits were primarily to myself. 
No immediate work-related relevance. 

A Great Deal: 

It was very beneficial for plafnirg the future. 

My field is advanced lithography now even though I had 
little previous background in this area. 

Gained initiation into vocabulary and many aspects of 
literature, current work, approaches to e-beam technology 
which win serve in good stead as basis in further, more 
detailed research. 
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Q. 49. How Much Did Your Company Bciefit ? (Cont'd) 

(A Great Deal): 

if I don't know this, I can't do my job. 



Somewha t : 
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When we make a decision on advanced lithography, it will 
be from an informed position. 

* 

t: 

The company benefits whenever I am able to expand my knowl- 
edge and/or meet others in the field. In this case, I 
established contact with 2 people who are new customers, 
and picked up background data on secondary electron genera- 
tion which eventually lead to a major decision in e-beam 
area. 

Nothing new was discussed. Some contacts with people at 
the meeting were useful. 

Useful information is beneficial. (Lower costs), shorter 
development time. 

Helped in selecting a system for purchase. 

Making me aware of problems and developments in other areas. 

Added background information to R 6 D work in progress. 

We are just getting into integrated circuit design, and it 
is useful to us to know alternatives. 

Obtaining list of attendees. Contact with interested parties 
who could become customers in the future. 

Ntost of lectures are reviews, but somewhat in current and 
practical technology. 

This will be more evident when we begin to follow through on 
' plans to set up an e-beam facility. 

This isn't a company. My research program benefitted. 

Very difficult to evaluate. 

Again w.r.t. deciding about pursuing e-beam lithography. 

Lithography is not my area of responsibility, so there is 
slight direct benefit. But my understanding of problems and 
opportunities raised by e-beam is an enhancement. 

Benefits still in future. 

General knowledge. 

Increased my knowledge of potentially applicable techniques, 
lias not yet been implemented. 

I coulc^'f absorb it all. Much of significance is beyond me* 

Q. 50. What one suggestion do you have for improving this course ? 

« 

3) Bverhart should have printed material of his opening remarks! 

6) Follow it up with lectures from different companies, and make it an 
ongoing, upgraded series, rather than a single session. 
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^^'hat one suggestion do you hive for improving this course? (Cont Ml 

11) I would have preferred more detail on e-beam systems. This, however, 
is the viewpoint of a person who works with an e-beam system. 

14) I thought it was about right. Perhaps more time at tiie end of the 
day to talk informally would have been good. 

15) Having a preliminary summary of fundamentals sent to participants 
before the class. 

17) Greater depth in less area. Treatment was too broad with the excep- 
tion of resist mechanism and dosage. 

18) Better notes. A well notated list of references. 

19) Too short- -need to be more thorough. 

20) More on real -world applications of this technology. 

22) Avoid repeating last year's material. 

23) Lecture notes could delve in to all areas more thoroughly than lectures 
themselves, rather than merely repeating. 

-24) Greater depth of coverage of current research and applications. 

25) Don't let a single company dominate the program. 

26) Start with the assumption that 50% of the audience knows nothing at 
all about e-beam. Build up slowly to the more complex material— two 
days might be better. 

27) Provide mare detail. 

30) Would like to have more accurate information on the quality and quantity 
of masks presently being manufactured on e-beam equipment. Not just 
isolated, but in a production operation. 

32) It is difficult because th audience background is so diverse. I 
think that you did about as good a job as possible. 

34) Mail notes to participants ahead of time. 

38) Break it into levels for the beginner and the experienced researcher. 

40) Better handouts at the course. 

41) More practical application details. 

42) Please completely separate fundamentals and tutorials from^current 
technology. Like the former in the morning and the latter in the 
afternoon. 

43) More discussion time--better organized. 
4b) Less IBM drcaras. 
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SUMIARf OF THE 
EVALUATION %f AIRPORT SYSTEMS & DESIGN CEE COURSES 
U.C. Berkeley, Engineering Extension 
(1978 and 1979 classes) 

Evaluators: Martha Maxwell, Ph.D^ 
Project Director 
Caroline Showers, M.S., R.A. 



The Airport Systems Planning and Design Short Course has three goals: . 
1) to provide a general introduction to airport planning; 2) to give an annual 
update on new inforipation in the field; and 3) to discuss new regulations for 
airport systems and the outlook for government funding. The four-day class is 
divided into four topics: planning and economics, operations, design, and 
environmental planning. 

This evaluation is based on the results of questionnaires administered 
during the last days of the 1979 class to 32 participants, follow-up question- 
naires returned by 21 of those attending the 1978 class, and interviews with 
the faculty coordinator and four participants. 

Characteristics of Participants 

The Airport short-courses attract students from a wide range of airport- 
related jobs*-planners, engineers, government officials, managers, architects, 
and other specialists. The average participant completed a college degree 
within the past ten years and has worked in airport-related jobs for about 
three and a half years. Twenty-five percent of the 1979 class had less than 
one year's experience in airports. However, in both classes there were some 
people with over 20 year's experience. Most « participants have administrative 
responsibilities and take the course to improve their general background and 
update their knowledge. Students in both classes report that the course met 
these goals well. Those who came for other goals (decision-making, meeting 
others, learning fundamentals) were somewhat less satisfied. Most participants 
felt that taking the course was valuable to them professionally, increased their 
knowledge, and gave them fresh insights. Those who had taken other courses 
rated the course as somewhat better than other short courses they have taken. 
The average participant was satisfied with the course. 

On the average, participants felt that the course level, and length was 
about right, although about a quarter felt the course was too fast, too short, 
and the content too easy. Participants rated the instructors as average to 
good, but judged the discussion and problem- solving sessions as weaker than other 
parts of the course. (Some questioned whether there were any problem-solving 
sessions.) Also a number complained that there was not enough emphasis on ap- 
plications. Note- some people attending the course had rather specialized in* 
terests such as expanding very small airports or building airports in the bush. 
It is difficult to address this range of interests in a lecture. Others wanted 
sitevisits and more emphasis on case-studies, and some were disappointed that 
there was not more state*of*the art information. 

This study was supported by the National Science Foundation under 
Grant «^SED-78-22136. The opinions, findings, conclusions, and recommendations 
expressed are those of the author, not of the National Science Foundation. 
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In both classes, some participants complained that there was not enough 
participant involvement in discussion or panels* Increasing the opportunities 
for participants to talk with one another and with the speakers is one way to 
meet the students' varied needs* * Also some complained about the length or the 
lectures and the length of the day while others in each class requested a 
longer course, (i.e., a 5-Oay course rather than 4 days.) 



1. Encourage those with special interests to form groups or meet others with 
similar interests individually (L .e. , those from small airports/ or those in- 
terested in design.) The course could be planned with rn opening social hour. 
More time could be scheduled for breaks (30 minutes vsi IS minutes.) Oppor- 
tunities could be given at the opening session for students to indicate their 
interests and find others with similar interests and ;jroblems. 

2. Plan the course to include more discussion groups (course topics, case 
studies, and problem solving) even if some of the formal presentations must be 
shortened. Consider reviving the panel discussion sessions, and schedule these 
during the early afternoon or morning if there's a problem with people leaving 
early in the late afternoon. 

3. As is typical of students in other CEE short-courses, participants in the 
airport courses show an increased interest in taking other short courses on 
the topics covered by the course. Some of this interest may stem from the 
course coordinator's deliberate trade-off between presenting general information 
on a number of topics vs in depth coverage of a few topics. U.C. Extension 
should consider extending this course or offering other airport short courses 
that would provide opportunities for narticipants to cover the topics in more 
depth. The topics that seem to be of most interest include financial management 
and planning, architectural design and construction, transit planning/multimodel 
planning, and environmental planning (essentially the main topics covered in 
this course) . 

Another way to provide interested participants who w<;nt more detail on 
some of the topics, would be to schedule some double sessions during the 4-day 
course where students would have to choose between two topics. 

4. There were a few comments that suggest some participants expected more 
emphasis on design. This seems to stem from the course title which, they felt, 
implied a 50-50 split between design and systems. Although the course descrip- 
tion clearly explains the emphasis and time devoted to each topic, the coordi* 
nator might consider how the title might be changed to avoid this problem. 



Recommendations 




erJc 



4 



C.5 P.l 



EVALUATION OF THE AIRPORT SYSTEMS PLANNING AND DESIGN SHORT COURSES 
U. C. Berkeley Engineering Extension 
(June 1978 and June 1979) 



Evaluators: Martha Maxwell, Ph.D. 

Caroline Showers, M.S. 



Sununary of Interview with Professor Kanafani, Faculty Coordinator , the 
Airport Syster^s course has three goals: 1) to provide a general introduction 
to airport ^)lanning; 2) to give an annual update, presenting what is new in 
the field; avA 3) to discuss new regulations for airport systems and the out- * 
look for government funding. One day of lectures is devoted to each of four 
major topics--Airport Planning and Economics, Operations, Airport Design, and 
Environmental Planning. 

The course is designed for professionals in a variety of occupations re- 
lating to air transportation. The group will include engineers, planners, 
architects, economists, and government officials. These people may be employed 
by airlines, airports, consulting firms, universities, and by the local or 
federal government. Some may also come ^rom overseas. The course is intended 
for professionals practicing in another sector of air transportation vAo want 
to broaden their knowledge to include airport systems and hence the emphasis is 
on introductory concepts. The annual update and discussions of regul^ations and 
funding may appeal to people with more direct experience of airport systems. 
It is expected that all participants will be college graduates who have majored 
in a technical field or economics and are now professionals in a field related 
to airport systems. 

The faculty coordinator anticipated that most participants from the public 
sector would probably work in local governments, as the federal government has 
its own training courses of this type. The involvement of government officials • 
in airport systems is usually from a client •s perspective—they have design or 
operations work done for them by a private organization. They may also play a 
supervisory role, which entails approval of airport plans or formulation of air- 
port regulations. 

The course has been given eight times. Over the years, the course foraat 
has been similar, however, each year two or three topics are changed to include 
current information. The few people who repeat the course do so after a four- 
or five-year interval. Usually, a company sends one or two of its employees 
each year, usually new employees. 

One benefit of the course to students is that they learn to design air- 
ports (e.g., runways, hangars, etc.) or to add new facets to their previous 
design perspectives. Design methods are published in the course materials which 
are very complete and include lecture texts, supporting reference articles, in- 
structions for design, and sample problems. The company benefits by obtaining 
these materials and also by the business contacts that are made. 




The instructional method is informal lecture with a number of visual aids. 
One or two lectures resemble workshops in that they demonstrate problem-solv- 
ing. Questions are permitted at any time. In the past, the coordinator has 
tried to end each day with a panel discussion, but he feels that this is not 
worthwhile because most participants prefer to leave earlier and explore the 
Bay Area. The only problem the instructor anticipates that students might have 
involves the breadth vs. depth trade-off. A wide range of material is present- 
ed, and, from the individual's point of view, some topics will be belabored and 
others will be passed over too quickly. 

The lecturers change every year and are selected personally by the coordi- 
nator because of their expertise on certain topics, .lost come from local con- 
sulting firms and many are former Berkeley students.' Topics are selected in 
terms of important factors of airport systems and current relevance (e.g. en- 
vironmental planning) . 

As mentioned above, the faculty coordinator stated that course notes are 
extensive and well^prepared, and are distributed at the course. No preparatory 
reading is suggested. 



Summary of Replies by 1978 Class 

TWenty-one questionnaires of the forty-eight sent were returned. ,The re- 
sponses seem to be affected by the year's lag between course and evaluation. 
There were fewer written comnents responses and there was a tendency to give 
average ratings. 

Sixty-eight percent of the group have Bachelor's degree, one is a Ph.D. 
candidate and the rest have Masters* degrees. They earned degrees over ^ 
35 year span with 27% having graduated in the past 4 years. Job duties show 
an even split between managerial and technical, typical of airport systems 
work. Sixty-eight percent of the^ group are either engineers or planners; there 
is one architect, 4 government officials, no economists, and one person who 
described himself as a contract negotiator. 

Fifty-eight percent have worked with airport systems for 5 years or less. 
The goals of: improving background and updating knowledge were rated as most 
important while fundamentals, perspectives, and meeting others were considered 
less important (rating 3). The achievement of goals shows a large variance, 
so that one cannot generalize from average ratings. Achievement is roughly 
satisfactory . 

Se\ sn persons expected to learn something that wasn't covered in the course 
and 3 had taken related courses. The course was rated better than others they 
had taken. Emphasis on research, fundamentals, applications, and theory was 
rated about right by most people. Significant deviations are that 2f>\ said there 
was too little emphasis on research and applications. 
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The various copies in the course were rated, on the average, between 2 
and 3 on a 1 to 4 scale of usefulness. A larpe Variance blurs any distinction 
aiqong topics. 

Twelve persons conmented on valuable ideas and 12 persons found the notes 
useful. Nineteen persons said the course helped them professionally to some 
degree and 14 persons suggested ways to improve the courses; 8 persons suggest- 
ed other course topics. 

The question concerning what topics in the course could be used immediately 
or in the future did not elicit many responses. Operations, Airport Design, 
and Environmental Planning were said to be useful immediately, while airport 
planning and economics were judged as useful in the future. Only 56% of the 
people returning questionnaires answered this question. 

The average grades for instructors ranged from B* to.C*. Grades assigned 
individual instructors showed a large spr'ad. Various aspects of the course 
were given the following average grades: Discussions, C; Social and economic 
factors, B-- or C*+; Fundamentals, B; and Applications, C*. 

Twelve people r^ted the course better than others they had taken, 8 about 
the same, and 4 less satisfactory than others. All biit 2 participants intend 
to do further study on the course topics as part of their jobs and \0 plan to 
study them on their own time. 

Eight-four percent anticipate discussing information learned in the pro- 
gram with their colleagues; and their supervisors. .Seven persons exp«:ct to use 
the course notes regularly, 14 think they might use them, and 1 does rot expect 
to use them. All but two said tJ.at they'd recommend the course to a colleague.- 

Three people wrote what they thought was most valuable and 69% thought the 
course helped them professionally. Twenty people made suggestions for improving 
the course and 11 gave ideas for other courses. Six made furthc: :?marks. 

[See Questionnaire following] 
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AIRPORT SYSTEMS PLANNING 6 DESIGN 1978 
^ Pollow-up Questionnaire 

nn^il!;* Motional Science Foundation has funded U.C.B. Extension to evaluate the 
quality and usefulness of Its short courses for engineers. Your responses to 
fh, 5 questionnaire will help us Improve our courses and develop an evaluation 
model for other courses. 

We appreciate your taking the time to answer these questions. 
5. What is your job. title? 

6-8. Your highest academic degree? vear r<.cel»/*rf? 

Aaim-M MA/Ks«b P).5«l received^ 

9-10. What $ of your time is spent in: »k-U«fc 
•^t^n* ( managerial duties'' WJ-*r*t 

* (.400-1 

IS*%1 technical duties H-lS - ^ 



[ other (what?) 



II. Do y^u work primarily as an: 

engineer 

3c planner 

[ f] architect 

[^] government offhe4al 

[O] economist • 

^ [/] other (specify) C«nWtftof\-Ys*j^o't\»:U^ 

12-13. For how many years has your work been related to airport systems? ' 

U-18. What were your goals in taking this course? Rank order. 
I ■ most important 
mm k * least important 

^•O [3 ] learning the fundamentals 

improving general background ^ 
lk.O updating knowledge 

acquiring a persp*!ctive for decision-making 
X.^ [\ ] meeting others in the field and finding out what they're doing 
( ] other (what?) 
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AIRPORT SYSTEMS PLANNING & DESIGN 1978 



Followup Questionnaire . 
• (N-21) 

The National Science Foundation has funded U,C.B. Extension to evalu- 
ate the quality and usefulness of its short courses for engineers. Your 
respones to this questionnaire will help us improve our courses and develop 
an evalxjation model for other courses. 

We appreciate your taking the time to answere these questions. 
5. What is your job title? 

6-8. Your highest academic degree? year received? 

B/A-14 MA/Ms-6 P.h.D-1 56 - 60-6 

61 - 65-3 

9-10. What % of your time is spent in: 66 - 70-3 

71 - 75-4 
76 - 79-2 



Mean « 


(47%) 


Managerial duties 




(50%) 


Technical duties 




(3%) 


Other (what?) 


Do you 


work 


primarily as an: 


\ 

40 


(8) 


Engineer 


30 


(6) 


Planner 


5 


(1) 


Architect 


20 


(4) 


Government official 




(0) 


Economist 


5 


(1) 


Other (specify) contractor/negotiator 



12-13. For how many years has your work been related to airport systems? 

1 2-5 6-10 11-20 21-30 

N» 1 11 3, 4 1 

14-18. What were your goals in taking this course? Rank order 
l»most important 
4»least important 

X Cranking it most important 

3.0 (3) learning the fundamentals 

2.1 (7) improving general background 

2.0 13) updating knowledge 

♦ 

2.8 . r3) , acquiring a perspective for decision-poking 

2.9 (1) meeting otht^rs in the field and finding out what they're 

( ). other (what?) ^ 

l6n 



lv-24. How well were your goals achieved? 





IIW W CI U 

all 


9WlU6 

what 


factorily 


very 
well 


learning fundamentals 


( ) 


(24%) 


(62%) 


(14%) 


updating knowledge 


( ) 


(10%) 


(55%) 


(35%) 


perspective for decision-making 


( ) 


(33%) 


(56%) 


(11%) 


meeting others 


(10%) 


(32%) 


(53%) 


(5%) 


25. Was there sometliing you ejq>ected 


to gflt 


from this 


course, but did' 


nt? 



7 or 33% yes, If so, what? 



26. Have you taken other cov ses on t^hese topics in the past 2 years? 
yes* 3 If so, what? 

Same type of course at Ga. Te^h. 
Refresher planning, FAA Academy 
Same type of course at MIT 

(MIT course better because it had a more diverse, 
interesting group of speakers. 

27. How did this c-urse compare with other engineering short courses you 
have taken? (in usefulness, content, interest) 

% 

(5%) the best I have taken 
(43%) better than others I have taken 
(19%) about the sames as others 
(5%) less satisfactory than others 
(28%) have taken no others 

28. How would you rate the instructorb^ emphasis on the following? 





too much 


about right 


too little 


current 'jesearch information 


( 0 ) 


(71%) 


(29%) 


fundamentals 


(10%) 


(90%) 


(0%) 


application 


(5%) 


(66%) 


(29%) 


theory 


(5%) 


(86%) 


(9%) 



\ 
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29-33* How useful have you found the topics covered? Rank Order 



l-most useful. 
4*least- in^ortant 

X 

(2,1) airport planning and ecomonics 

(2.5) airpcrt operations / 

(2.3) airport design 

(2.8) environmental planning 



34. What ideas and techniques have you found most valuable? 

gEE LIST 

35. Have the notes/course materials been useful in your work? 

Yes 68.1 No or haven't used 32% 

36. Do you feel that this course has helped you professionally? 
If sOr what? 19 said yes, 1 said no. 

37. What suggestions do you have for iniproving chis course? 
14 people made suggestions | 

38. What other short-courses would youl like to take? 

8 people made suggestions 1 



Other remarks? 



\ 

\ 



\ 

\ 



\ 

\ 



\ 
\ 

\ 
\ 
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Summary of Responses of 1979 Class 



Forty- six persons registered for this four-day course and 32 completed 
questionnaire? ai the end of the course. 



Characteristics of Participants 

With one exception, the respondents were college graduates, including 9 
Master's and 1 Ph.D. Sixty-three percent had received their degrees in the 
last ten years, making this a relatively young group. The average respondent 
spends 40% of his time in managerial duties and 51% in technical affairs. Ten 
people mentioned other activities such as teaching, public duties, and field . 
assignments. The participants included 14 planners, 13 engineers, 5 government 
officials, 2 architects, no economists and 5 "Other:*" (2 tea'^hers, 1 field . 
maintenance, 1 safety, 1 airport operations). Twenty-five percent had less 
than one year's experience in arport systems, and 72% had less tl an five years' 
experience. 

Participant Evaluations ^ 

Participants ranked their goals for this course as follows, from most to 
least important: 1) improving general background; 2) updating knowledge; 3) 
learning fundamentals of a field new to me; 4) acquiring a perspective for 
decision-making; 5) meeting others in the field. One person mentioned business 
contacts and ranked this goal third. It appears that most participants do not 
consider themselves "new to the field," but are interested in improving their 
backgrounds. The first three goals were found to have been met satisfactorily, 
and the latter 2 were judged as being "somewhat" satisfied. 

Twenty- five percent said that there was something they expected to learn 
in the course but didn't. Most people found the pace, length, and level of the 
course to be a'jjut right. However, 19% said the pace was too fast, 28% said the 
length was too short, and 25% said the level was too easy. 



(For details, see Questionnaire following) 
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AIRPORT SYSTEMS PLANNING & DESIGN - June 18-21 
June 1979 Class 
Questionnaire 
N«32 replies our of approx. 46 registrants 



The National Science foundation has funded U.C.B. Extension to evaluate 
the quality and usefulness of its short courses for engineers. Your responses 
to this questionnaire will help us improve our courses and develop an evalua- 
tion ttiodel for other courses. 

We appreciate your taking the time to answer these questions. 
5. What is your job title? (See p. 4.) 



6-8. Your highest academic degree? 

Less than BS=1 BA/BS»20 MA/MS=9 P.h.D.=l Year received? 

50 - 55=1 

9-10. What % of your time is spent in: ^ 56 - 60=- 

61 - 65*4 

Avg= (40%) mancgerial duties 66 - 70=4 

(51%) technical duties 71 - 75*12 

(9%) other (what?) 76 - 79«8 



0» 

11. Do ycu work pririuirily as a 

(36%) planner 

(33%) engineer 

( 6%) economist 

(5% ) architect 

(13%) government official 

(13%) other (specify) 

2 teachers field Airport operations 

maintenance safety 

12-13 • For how many years has your work been related to airport systems? 

1 2-b 6-10' 11-20 21-30 31-40 

8 15 4 2 2 1 

14-19. What were your goals in taking this course? Rank order the following 



ERIC 



uoing #1 


for the 


most important? #6 for least. 


Avg. 
3.0 


# listing each as most important 

(8) learning the fundamentals of a field new to me 


1.5 


(15! 


improving general background 


2.1 


(9) 


updating knowledge 


3.7 


(1) 


acquiring a perspective for decision-making 


3.8 


(1) 


meeting others in the field and finding out what 




( ) 


other (specify) contacts-^ ^ 
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20-26. How well did the course meet /our goals? 



not at all somewhat satisfactorily 



% % 


% 


% 


• 

learning the f und2unentals (3%) (30%) 
improving background (3%) (10%) 
updating knowledge (6%) (19%) 
acquiring a perspective for 

decision making (13%) (60%) 
other? ( ) ( ) 


(50%) 
(57%) 
(52%) 

(37%) 
( ) 


(17%) 
(30%) 
(23%) 

(13%) 
( ) 


27. Was there something that you expected to learn in 
if so, what? 8 persons said "yes" (25% of group) 


the course, but did'nt? 


28-30. How would you rate the course on the following? 

% % % 
pace (19) too fast (24) about right ( 7 ) too slow 
length ( 7) too long (64) about right (29 ) too short 
level (26) too easy (71) about right ( 3 ) too hard 




Did you find anything in the 
course that you could use 
immediately? If so what? 

# 


Did you find emy thing 
course you can use in 
ture? If so, what? 

# 


in the 
the fu- 


Airport planning 4 
&economics 


7 




Operations 7 


3 




Airport Design ' 8 


5 





Environmental Planning 



10 
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(On the reverse side, briefly describe the ideas and techniques you found 
most valuable and how you plan t6 use them.) 3 persons commented 



How would you grade the instructors on content and presentation. Use A« 
F^Very Poor 



AvQ. Grade 



Antis 


IC+) 


Brawner 


(B-) 


Doyle 


(B+) 


Galloway ^' 


(B-) 


Gosling 


(B ) 


Hockaday 


(B+) 


Howard 


(B ) 


Kanafami 


(B ) 


Joerger 


(B-) 


Pollack 


(C+) 


Robert 


(B-) 


Ashford 


(B ) 


Wesler 


(B+) 


Whitehead 


(B-) 



Paul (A-) 




ellenti 



45. Rate the following aspects of the course using A^-Excellent through F=Poor. 

Averages (C) discussion and problem solving workshops 
(B-) social and economic factors 
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Averages 

(B) fundamentals (theory and research) 
(C+) applications 

46, How did this course compare with other engineering short courses you 
have taken? (in usefulness, content, interest) 

% 

{ 0 ) the best I have taken 
( 18) Better than others I have taken 
( 25) about the same as others 
( 12) less satisfactory than others 
( 25) have taken no others 
N+32 

47. Do you plan to do futher- study on the topics in this course? 
(17%) yes, as part of job 

(2 3%) yes, on my own time 

( 0%) doubt that 1*11 do more study 

( 0%) am sure I won't 

43. Do you plan to discuss the information gained in this cours* with your 
company colleagues? (73%) supervisor? (73%) 

49. Will the course materials be useful to you in your work? 

(31%) expect to use them regularly 
(64%) may use them 

(5% ) doubt that I'll use them N»22 

50. Do you think that taking this course will help your professionally? 
If so, how? Yes»22 or 69% 

51. Would you recommend this course to a colleague? (7%) No (93%) yes N=30 

52. How might this course be improved? 20 people made suggestions 
5 3. I would like to participate in a short course on: 11 ideas 
Other remarks? 6 comments 



The NSF Evaluation Project Team is planning to conduct case-studies on engineers 
who have taken short-course? . If you are willing to participate in two brief 
interviews, to be scheduled at your convenience, and to permit us to interview 
your colleagues and supervisor, please fill in your name and phone number below: 

Name 



Phone No. 

(area code) 

( ) prefer telephone interview 
( ) want further information 
( ) do net wish to participate 
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AIRPORT 



Participants' Suggestions for Improving the Course 

Content 

More talks of how to improve an existing airport and facilities and not 
how to ^ild a new one. There are very few new airports in the world^ 

Too much emphasis on planning; split planning and design more equally. (2) 

Extend course to cover more design considerations, or separate it into 
two dif.^'erent courses^ one for planning, one for design. 

More practical planning approaches from the airlines' point of view. 

More discussion on problems of smaller, growing airports--not just relat- 
ing examples, to major world airports. 

More emphasis on multimodal--and an airports system planning, i.e., a 
system of airports. 

Cargo lecture is too specific--should be restructured to focus on how 
cargc fits into the entire system--financially, operationally, etc. 

Speakers 

It would be difficult, but all instructors should review each other's 
notes so that repetition and overlap can be avoided. 

flavc speakers add index for abbreviations in their notes to eliminate 
student confusion. 

f Six topics were presented by consultants. Five of the six were by P.M.M. 
This gives the appearance that they have the only experts available and 
places other consultants at a disadvantage. 

Try to provide equal representation from consultants who give presenta- 
tions. P.M.M. was overly involved in program. 

Improve quality of presentations. 

M<;vc more professionals for presentations and fewer academicians should be 
involved, as they are out of touch with the realities of the business. 

Most speakers were incredibly boring. Perhaps a different format should 
;:e tried. Panels were better than individual speakers. 
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Speakers (cont M) 



Broaden the range of speakers- -seemed like too inbred a group Increase 
participation by practitioners (the airline executive who participated 
in fleet planning was good.) Gosling's stuff was much too basic. Kanafani 
needs to add life to presentations. Environmental presentations were 
useless. 

Speakers should refrain from speaking on the subject matter only as it 
relates to their airport'. 

Your expert speakers were obviously well-qualified in their fields, but 
they were apparently not too excited about participating. Please don't 
teach FAA manuals. 

Put more emphasis on new techniques, ideas, etc. 

Hliminate complicated subjects unless time is allowed to properly address 
the subject. 



Discuss ion/ Prob 1 em- So 1 v ing 

Participants can learn mucji from each other. Encourage contacts. 
Increase the discussion sessions (3) 

Would like to see additional problem solving sessions. (2) 

More case studies. (4) 

Develop problem-solving atmosphere. 

Discussion on the effects of deregulation. 

Organization/ Scheduling 

Organization of seminar overall was good. 

lengthen course to 5 days with longer lunch. Find a decent hotel (other 
than Motel Durant) . Provide more information on San Francisco (eating, 
entertainment). Schedule more social gatherings- -luncheon or dinner with 
guest speaker or informal activities. 

Daily lecture schedule rather long. 

Extend course to full week. (3) 

5 to 8-hour days \.oo short. Evening sessions would be appropriate to 
hold it to a week. 
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Organi zat i on/Schedul ing (cont 'd) 



Fewer topics/greater detail. 

Increase the amount of topics, the amount of subjects in each topic 
and discussion sections. 

Limit lectures to one hour between breaks. Need classroom work with 
field trips appropriate to the course and location. Bay Area has much 
to offer those of us in less sophisticated environments in terms of 
facility experience. 

Course Materials 

Loirplete list of bibliographical materials in these fields; brochures, 
pamphlets, pictures--All should have been given out as a package. 

Include some case studies on handout. 



Responses to Q. Describe the ideas and techniques you found most valuable 
1979 Class 

Because each topic was necessarily general in nature, I believe the most 
important aspect was in providing references to both design and planning 
texts, but also to individuals and companies in the field. 

I got the impression I'm on the right way toward doing my job! 

Terminal cargo--good "rule of thumb" technique useful in landside 
planning. 

Economic planning--provided good background on the economic operation 
of a larger airport. Will help one provide advice to clients. 



1978 Class 

Design and operational techniques. 

Those relative to facility planning (as opposed to land use and runway 
planning) . 

All have assisted in refreshing my knowledge of various aspects of 
planning. 

General background 

Design for flexibility. i 7*2 
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1978 Class (cont'd) 



Airline planners* discussions of operational needs and planning criteria. 

Airport management in face of deregulation; environmental impact state- 
ments. 

Airport planning and projection of needs. 

Perspectives of Washington- level environmental people and air cargo 
managers were helpful. 

Design of apron and terminal facilities. 

Forecasting the demand for air transportation. Also, environmental and 
noise sections. 

Recognition of various perspectives of noise problem. 

The idea that there arc several points of view on any problem. 



How Course Helped Them Professionally 

The most frcqiient answer was "By broadening my background." Some other 
examples: 

The major benefit of this course is to become more knowledgeable in some 
areas so as to aid in professional interactions with individuals in those 
fields. 

Currently involved with many issues covered in the course. 

In gaining a more general knowledge of airport planning and operations. 

Update information; identify sources of additional information*-*-notes 
and references. 

Introduction to new work area. 

In planning and administration, coordination and problem studies* 
Gave mc insights into planning and design needs of airlines. 
To keep up with rapidly changing field of noise evaluation. 
Fundamental technical knowledge. 
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It brought me up to date and refreshed my knowledge of the industry. 
Perspective and personal contacts. 

Gave mc good concise information that I can use everyday. v 
What Tliey Expected To Lcarn^ But Didn't 

1979 Class 

Less theory--more practice 

Course has too much "what" and not enough "how" . 
Specific detailing on design 

Much of course was a review type approach. I needed more fundamental 
material . 

I expected more precise definitions, examples of some aspects of this 
course. 

Alternative airfield design concepts. 
Site location decisions. 

It there was a national scheme for developing airports. It seems this 
is a hodge-podge industry. 

Would have enjoyed a little more direction toward general aviation 
activities and operations. 

1978 Class 

More emphasis on multi-modal transportation planning. 

New ideas or techniques. 

( 

A little more 'hands-on' analysis exercise in, e.g., capacity or gate 
occupancy. 

More specific case study situations. 

More student input. 

Planning 

More state-of-the-art information--i .e. , forecasting advances. 
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Responses to Q. 53. "I would like to participate in a short course on..." 

(1978 § 1979 groups) 



Financial management, plannirig;' airport economics with emphasis on cost-effective 
approach to design, construction, and operations. (5) 

Airport architectural design and constructions; actual design applications; 
aviation facilities (2) (as related to bush and developing countries (1) 
(Total - 5) 

Airport environmental planning, growth and management in land use 
planning (3) 

Transit planning: engineering education/urban transportation planning and 
policy ana^lysis; multimodal system planning (3) 

General aviation (3) 

Airline and airport management and marketing (2) 

X'hose that might examine individual items like runway capacity in greater 
detail (2) 

\ 

Forecasting (1) 

Lease agreements on airports; financing alternatives, airspace capacity and 
analysis (1) 
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TH2 £;<gCN£NTIAL SMCOTKINC AND A DAPTIVE FCRF.-A.c;TTNr. 
TgCH^I^U-S COURSE OF JUNE 27, 1979. U.C. Berkeley, Continuing 
.Education m Engineering. (Prof. Robert M. Oliver, Faculty 
Member in Charge . ) 

Evaluatorst Martha Maxwell, ?h.D. 

Carolin Showers, M.S. 

Nineteen of the 26 students enrolled in'.the Exponential 
Smoothing and Adapt ivg__Forec/s ting GEE Course (June "27, 1979) 
returned follow-up questioftKaires. Although the 53 percent of 
the class who responded may not be- representative of the total 
class, their responses suggest that the class was widely diverse 
in regard to preparation, experience, and expectations for the 
course. Considering the heterogeneity of the' group, ■ it is not 
surprising that student evaluations of the course were mixed. 
(See Appendix attached fbr response distributions.) 

Six students (32% of the class) said. they had taken no other 
courses in forecast^ , while 13 had. However, some' of the latter 
group wanted^ and (i.. i rently) needed an intensive review of basic 
fundamentals so thar even if admission to the' class were made 
contingent on students' completing basic courses in forecasting, 
the cl^ss would still contain pedple with diverse needs and there 
would be a good chance that it would not atvtf'act enough students. 

What this diverse group did have in common was an interest 
in the subject and the motivation tc learn more about the field. 

Teaching short courses to such a varied group' of " stiidents 
poses many problems for the ins true tor- in-c har ge The following 
are some teaching strategies used by other instructors in organizing 
and planning such* inf ormation-intensive , one-day engineering short- 
courfies where students var/ widely in background and experience. 

--Alert students in advance of the courge to some basic 
materials to review in preparation for thp cpurse. These 
might include a ^ textbook or summary articles. The easiest 
way to do this is to include references in the course bro- 
chure (or students could bis, sent a list before the course.) 
Not all of the students will read materials in advance 
^ . references can be very helpful to some students. Others 

won t review the materials before coming to the class, but may 
be inspired to read xhem afterwards. 
-- Plan a brief (about 1 hr.), thorough introductory lecture 
covering the basic concepts in the field and open the class 
, with this. Explain that this information will serve ak a 7 

review for the more advanced students, but will help the"^ 
others get a foundation for the rest of the course. (Advanced 
.> students are usually tolerant and accept this kind of explan- 

ation. ) 

— It is essential that students be given a rather complete 

outline of this introductory lecture that includes the topics, 
key terms and their definitions, formulae, main concepts, 
and some references. Underprepared students are less 
critical of courses where they are given notes that they 
can review afterwards, and , in this course, the majority 
indicated that they planned to do further study on the 
course topics. 

Increasingly, engineers who attend CEE courses expect 
good, well-organized, course notes to be distributed by 
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instructors and notes mi^e it easier for the instructor 
to cover tHe course objectives in a limited time. 
■Introduce each coui*se. topic by presenting carefully selected 
examples or problems, discussing these, and then presenting 
the underlying theory. Discussing examples first reduces 
the frustration o^the less sophisticated stjidents while 
maintaining the interest of the more advance,^ ones. Students 
respond well to what they consider real-life examples 
or problems*. 

■If possible, organize course around projects (i.e., have 
students send in descriptions of their projects or collect 
exsLmples from students who have taken the^ourse before.) 
■Set limits on the number and kinds of questions students 
may ask. Some^ instructors answer only questions involving 
clarification during' their presentation, deferring discussion 
questions until the end of the session (or end of the day.) / 
If students are allowed to debate speculative issues it caTi / 
take a lot of class time and alienate the rest of the class 
who feel left out. (Some Berkeley students are experts in 
diverting the instructor's attention from the course topic 
by their questions.) 

Allow enough time during coffee breaks for students to 
talk with each other and those who have questions to talk 
, to. the instructor. CWotei<-15 minute bre^BOcs are oftten too 
short— 20 minutes are better, if they can be arranged.) 
CE£ classes often vary in composition from one time to another. 
If a few questions are included on the course brochure 
for students to fill out amd send in with their registration 
fees, the instructor can get,t some ad vane e information about 
the kinds of people who'll be in the class. For example, 
questfons on demographic characteristics, interests, and 
experience in using varioUs forecasting techniques, 
ty^^es of porjects involved in.. could be asked. This information 
would help the instructor in planning the course and also , 
help the students recognize that the instructor is aware of 
their needs. - ~ ' 



17', 



June 27, 1979 



EXPONENTIAL SMOOTHING & ^OAPTIVE FORECAST'^NG 

TECHNIQUES FOLLOW-UP QUESTIONNAIHE 
{N-19 Questionnaires retained out of 36 sent) 

The National Science Foundation h^s funded U.C.B. Extension to evaluate the 
qwil.'ty and usefulness of its engineering short courses. Your responF^s to 
this questionnaire will helt* us improvri our courses and develop an evalua- 
tion model for other courses. 

We appreciate your taking the tiw^^ to answ^ / these questions. 

%. What is your job title? See page iv, 

6-8. Your highest academic degree? 

BA (2); BS (5)? MBA (2); MS (6); P.h.D. (3) Year received? 

* -10. What % of your tire is spent in: AVERAGE 

( ) managerial duties 20% 
( ) technical duties 80% 
( ) other (what?) (marketing mentioned by 2) 

11. Do you work primarily as a 

( 7 ) corporate planner Other? (What?) 12 

12-13. For how many years has your work involved forecasting? 

0-2; 1»1; 1-2 yrs.«6; 3-5 yrs«5; 6-9«2; 10+-2 

14-26. What were your goals in taking this course? Rank order the following 
using 41 for the most ^.mportant? #6 for least 



{ )• Learning the fundamentals of a field 

new tc me 

* 

{ ) improving problem solving ability 
( ) updating knowle^ilge 

( ) acquiring a perspective for decision- 
making 

{ ) mating others in the field and 
finding out what they're doint 

( > improving general job skills 

.16^ (specify) 



How well were your goals achieved? 

no . 
resp . 
(2) 11% 



not at 


some 


very 


all 


what 


well 


(6) 31% 


(11) 58% 


(0) 


(4) 21% 


(9) 47% 


(1) 5% 


(1) 5% 


(13) 67% 


(3) 16% 


C3) 16% 


(10) 53% 


(3) 16% 


(9) 47% 


(4) 21% 


(0) 


OJ^6% 


(9) 47% 


(2) 11% 








{ ) 


{ ) 


( ) 



(6) 32% 



( ) 
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27. Was there something \,hat you expected to learn in the course, but 
didn't? If sor what? 

See P.V. 

28-30. Please rate the course on the following? 

pace (4) too fast (8) about right (7) too slow 

length (3) too long (5) about right ' (1^1) too short 

level (2) too easy (9) about rigtt (8) too hard 

31. Please describe the ideas and techniques you found most valuable 
and how you ^lan to use them. 

See P. VI. 



32. 


Rate the 


following 


aspects 


of the 


cotirse 


using A=Excellent through 








A 


B 


C 


D 


F 


No answer 


( ) 


fundaunentals 


4 


2 


7 


1 


3 


2 


( ) 


theor 




3 


4 


5 


1 


4 


2 


( ) 


appi ^ - . 


.13 


6 


3 


1 


2 


6 


1 


( ) 


Audio- - 


a1 systel 


11 ' 


5 


1 




1 


1 


( ) 


use of 


puter 
















termina: 




10 


4 




2 




3 



33. ^Have you zaken other cotirses in forecasting? Yes»13 No«6 

! 

34r • How does *^^his course compare with other engineering courses you have ta^ en? 
(in ^^sef ulnes< , -ontentr interest) 

{9^^o, best I have taken 

(3) better than others I have taken 

(3) about the same as others 

(5) less satisfactory than others 

09) have taken no others 

35. Do you plan to do further study on the topics in this course? 

(12) yes^ as part of my job 

(9) yesr on my o%m time 

(3) doubt that I'll do more study 

(Ojx^ sure I won't 

/ 36. Have you discussed the information gained in this course with your 
/ company colleagues? (17) si^ervisor? (15) 

37. Will the course materials be usefvl to you in your work? 

(2) expect to use them regularly (13) atay use them (3) doubt that I'll use 

them 

(1) expect to use, but not reguxariy 

38. How might this course be improved? 

See P. VI 
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39. I would like to r^articipate in a short course on: 

S«e p. w.i, 

Other remarks? 



The NSF Evaluation Project Team is planning to conduct brief phone interviews 
with participants of this course to assess the usefulness and appllcabU ity 
of the information covered. If you are willing to take part in a brief 
interview, to be scheduled at your convenience, and to permit us to interview 
your colleagues and supervisor, please fill in your name and phone number 
below: 



N.aaje 



Phone No. 



(area code) 

( 1 want further information 
( 1 prefer telephone interview 
( 1 prefer that you contact: 



Phone 



Job title 



[ 1 do not wish to participate 



Return to XT 37 Engr. Extension 
2223 Fulton Sc. 
Berkeley, C\ 94720 
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Forecasting 



Replies to 5- ^at ypur joh titip o 

— or. Programmer/Analyst 
Financial Analyst 
Systems Analyst 

Sr. Assoc. Systems Analyst 
Director of Analytical Services 
Senior Operations Research Analyst 
r/.arketing Analyst 
Planning Analyst 
Manager Manpower Planning 
Sr. r..arket Planning Analyst 
Assoc. Civil Engineer 

— Analyst/ }-rogrammer 

— Administrative Analyst 
Forecasting and Planning Analyst 
r^'.anager of Planning 

— Assistant Dean 

— iVathematician/Programmer 
Senior Associate (Consultant) 
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Responses to Question! Was there something you expected to learn 
in the course, but didn't? If so, what. 

—reason for using exponential smoothing model vs other techniques. 

--I thought the course would concentrate more on different formulations 
of adaptive forecasting. 

—More practical discussion of fprecasting techniques and use of 
the computer. 

— Specific model formulations. 

— A factual, non- theoretical approach to forecasting. 

— A practical and simplified method for doing forecasting via 
exponential smoothing. 

Hands-on experience. 

— Exponential smoothing. 
No. 

— How to actually apply the theory. 

— Expected to see a more balanced approach between academic and 
professional presentati'^n perspective. Too much Greek/ Could 
have been done in a more verbal context much better. 
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Responses to Questioni Please describe the ideas and techniques ;ju 
' found most valuable and how you plan to use them . 

— Smoothing; & forecastinf^ of low data rate (1 sec. > sample "^15 mins. ) 
Instru.'nentation systems. 

Have already used double exponential smoothing to arrive at a 
forecast based in extensive professional situations. Find it 
most useful over very large sample bases of sound data - a rare 
practical occurance C*^; 
— Not applicable. 

1 can apply EaTv.a Technique and Holt-.Vinters Method, maybe adaptive 
forecasting^ to my new responsibility. 

— Effect of smoothing constant. 

— Zip. 

— Forecasting bookings & billings via exponential smoothing. 

-- Basic ideas substantiated knowledge already acquired and reinforced 
decision to use these techniques. 

— The general concept of exponential smoothing using trend and 
seasonality is directly related to the forecasting software 
package we use. 

— Basic formulations may help in addressing forecasting problems. 
Not sure how far 1 could get without further reading, however. 

— The adaptive f orecastin,; part was valuable. However, it wat too 
short. 

— Lagged autocorrelation (?) 

Hesponses to Question; How mif:ht this course be improved ? 

— Applications requiring judgmental ass-'ssment of model. ' 

— In general, the course was too easy for me. I w. uld prefer a more 
theoretically oriented course. 

-- N'.ore practical discussion of forecasting techniques and use of the 
computer. 

I^.ore handouts. 

-- Give additional time to forecasting fundamentals. Get away from 
theoretical to practical examples. Use real data or participant 
supplied data as examples. Have students participate more need 
more of a workshop atmosphere. Definitely provide more time . 

-- More practical. More organized. Faster paced and shorter. 

— More theory. Prerequisites for admission to course. The prof, 
should prepare a lecture *. J S3 

ER?C — Keike clearer which parameters are derive*! (and how they are 

■ iffljiiteM derived) and which are estimated (& the bases for the values chosen*) 
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How mif^ht che course be improved ? ( Cont • d ) 

The course was not aimed at me. It was too technical and too 
quick. By the time I had assimilated material we were long past it. 

— ^:o^e down-to-earth explanations — too much theory. 

Due to the mixed-education background and work experience among 
the audience, it seems lacking the interation between the instructor 
and the audience. I would like to see the pre-req. be enforced as 
the condition to enroll in this course. 

— Take it out of the theory level and discuss it in layman's terms. 

— Take out the Greek and use Enf^lish languan;e descriptive nouns . 

-- Kore exposure to smoothin^forecastinc technology from signal 
processing theory. 

^.ore time on the computer. 

ether remarks ? 

-- I can justify a weekend short course to myself easier than weekday 
courses to my company. 

— Keep me posted on course offerings. - 

-- I have had 3 graduate level statistics courses from a psych, dept. 
and ain MbA which also required statistics. The course description 
said that a person should have some familiarity with statistics. 
I even called the instructor who said that as long as I understood 
standard deviations and correlation, I*d be fine. Well I wasn't. 
I wasted a day and the tuition. 

The course should have been billed as one for people actively 
using exponential smoothing methods' or (those with ?) advanced 
statistical backgrounds. The pace was much too quick for someone 
who was unfamilar with the topic. It was not broadly aimed at 
general business problems. I*m sura an analysis of questions asked 
would show a great prep^onderance revolving around academic oriented 
technical questions (both in numbers and time spent) rather than 
applications or problem-solving questions. The course was pro- 
bably of value to some people but not to me. As a result I 
will certainly be more cautious in selectii-ig or recommending 
future options. P.S. 1 might add that this course was described as 
being for business people and not exclusively for engineers. 
P. P. 3. The instructor made poor use of the time he had. ror 
example, he got a late start and then got hung up in the descrip- 
tion of his multi-colored syllabus and the rest of the day he was 
pushing trying to make up lost time. 

As I get worked up again ( I had mellowed out since June 27 
I remember one of his statements early on where he decided not co 
cover some very basic information like when or how should I 
forecast, what is special about my data, etc. He said the material 
had been covered in a previous courses and that there were some 
familar faces so he wouldp'-^ both'^r to go into it again. I was 
Q not a familar face. 
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- The course was adverticjoH oe a- 
a forecastinc system^or the non!e^fJeer. " "'^ "''^ 'or 



" Z lolTng inS?Me"^: P^-'-tion. filter. 

- Keep me posted on course offerings. 

" ^^'y^'^ ( i-e. Box-Jenicin 

- Basic forecasting. 

- ether forecastinc techniques- denographic - economic. 

- technoloey forecasting (substitution, growth curves, etc.) 
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Background & Purpose of the Course 

The National Information Service for Earthquake Engineering 
(NISEE) , sponsored by the National Science Foundation (NSF) , 
distributes computer programs for earthquake analyses of 
structures. These computer programs (or codes) are developed 
at uni/ersities (primarily at U.C. Berkeley) under NSF grants. 
Codes can be purchased by the public from NISEE for $125-$250. 

The purpose of this course is to describe four computer 
codes for earthquake analyses, to relate practical experience 
in their use, and to demonstrate the problems for which they 
can be successfully applied. The course serves both as a way 
to advertise codes to engineers, especially those in small 
companies who are unfamilar with them, and to update the under- 
standing of engineers in larger companies who have already 
purchased and are currently using the codes. The course 
coordinator made special efforts to encourage present o%mers 
of the codes to attend so they could improve their understanding 
of the types of problems for which the programs should and 
should not be used. 

The coordinator expects that participants in the course 
will have varied experience and background. Most are expected 
to be structural engineers, involved in the design and analysis 
of buildings who have had some experience with computer applications 
in structural design but not specifically with programs geared 
to earthquake stress analysis. The coordinator anticipates that 
architectural engineers and state and city engineers who oversee 
building design will also attend. 

In order to profit from the course, the instructors expect 
participants to have some background in fiondamental building 
analysis, at the level of an undergraduate course in dyitamics. 

The six speakers were chosen because they participated in 
writing the codes taught during the course. Four are from 
academic backgrounds — two University of California Berkeley 
professors (Wilson and Powell) and two of their former students 
(Row and Rollings) . The other two instructors are practicing 
engineers who have a great deal of experience in applications of 
the codes (Habibullah and Fintel) . 

The course is organized in four parts by the computer 
codes: ("ABS, ETABS, DRAIN 2D and DRAIN-TABS) Instructors 
introduce each code, give examples of its applications, and 
discuss the limitation of its use in certain problems. 

Input manuals for each of the four codes, including a brief 
explanation of the theory for each, comprise the course materials. 
In addition, materials showing actual examples of how to use the 
codes are given but. 
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After attending the course, participants are expected to 
gain experience by using ;he programs in their design work, 
gradually learning the types of problems to which the codes 
can be successfully applied. 

Results of the Evaluation 

Questionnaires were passed out to participants or^'the lajt 
day of the course and returned by 49 people of the 50^ho 
enrolled in the class. Eleven students were intervi4wed by 
phone one to two months later. 

Student Characteris,tics 

The studeny^oup had very applied interests and work 
primarily in structural design and analysis. They are fJtirly 
well-educated, With degree level split evenly between Baclielor's 
and Master's with just 3 Ph.D.'s. Most (63%) received theit 
degrees after 1965 and spend more than 90% of their time in 
technical duties. Half say that their work has involved earth- 
quake engineering for more than 5 years. Most had performed 
computer analyses before t iking the course and applied computer 
results to building design. Forty-three percent have, written 
applicable computer codes while 10% had no previous experience 
with computer progreuns. 

Jtost people came on th^ir own initiative, while 24% came 
at tn9ir supervisors' suggestions and 18% were selected by 
their companies. Eighty percent had their expenses paid by 
the company, while 42% received released time, and 27% noted 
that their companies strongly encouraged attendance. 

Student Evaluations 

Most participants say they took the course to learn and 
update their information on computer applications. Students 
felt that the goals of updating applications and meeting others*^ 
were well satisfied. Those who wanted to learn new applications 
and gain perspectives for decision-making were somewhat less 
satisfied with the course. Most people were interested in 
the TABS and ETABS programs, and a majority of persons felt that 
they could use information gained about these immediately. 
Fewer people felt they would use the DRAIN models immediately 
although the majority said they might use them in the future. 

Seventy-two percent said the course pace was about right, 
while 21% found it too. fast. The instructors' emphasis on 
funtdamentals, applications, theory, and research was judged 
about right by most people. The only significant deviation 
was that .30% felt there was too little emphasis on applications. 



The average attendee said he might use the course materials 
while 39% said they plan to use them regularly. 

Most (82%) felt the course had benefited them professionally 
and that their companies would benefit somewhat from their taking 
the course. They gave a long list of reasons. 

Half of the group plan to taJte additional courses. Most 
plan to discuss the information with their colleagues and, about 
half will talk to their supervisors about it. (Most students 
(87%) said they would recommend the course to a colleague.) 

Thirty five people made suggestions for improving the 
course. The most frequent comments were on the need to make the* 
course more applicable for practicing engineers (e.g., more 
information on the pros and cons, costs, etc. of different 
programs) and increasing the length of the course (e.g., following' 
it with a hands-on workshop) . Also suggestions were made on 
improving the materials (e.g., including sketches in the hand- " 
outs, sending materials out in idvance of the course, etc.). 
Some indicated they had problems following the talks of some of 
the lecturers (e.g., they jump about too much). 

Information from some of the interviews illu8tr*te^ the 
different ways participants viewed the course and how they are 
making use of the information they learned. r 

Case #1 - a president of a structural consulting fLjsirl^ho liad 
used a variety of programs for ' structural analyst^; but»was 
not familar with quake modeling before taking this course. 
He feels his knowlc^dge has improved, he has used TABS twice, 
and hopes to use the programs more because he finds them easy 
to use. He has been designing high rises for a long time, so. 
his mode of design will not change, but the computer programs 
will assist in making preliminary analyses. He feels these 
programs are the first that are cheap enough to use in actual 
design process and are more efficient time-wise compared to 
long-hand calculations for a 30-40 story building. He states he^ 
will probably go ahead with using a static loading but thinks a 
dynamic loading would give greater deformation althbugh he's 
not too confident about the use of dynamic loading as input. 
He's used the course notes only for TABS and says using DRAIN is 
premature. He felt the instructors in the cburs^ gave impressive 
presentations and appreciated the discussions^ of computer methods 
when long-hand calculations were presented side-by-side with 
computer programs. He needs this to justify the use of the 
programs in consulting. *. 

He feels the DRAIN (inelastic) progreuns are not useful because 
the physical processes are not well understood and clients would 
not see the results as beings reliable.' He sjl^ys the main benefit of 

'the course was to get programs w^ich he can use economically. He 
thinks the course could be im(h;pvi»d if there w^r« work sessions for 

individuals to run sample prdgra/ns themselves. 
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Case 12 - d director of research aijd development who felt he 
did not have much preparation for 'the course but wanted to know 
how to use the programs. He feels he gained a lot of information 
but found too many things were missing in the programs. Since 
taking the course he has tried to set up a program but found it 
impossible. He drove the data to the nearest university where 
the computer department told him the job was impossible to run 
on their machine. He's upset that he wasted material and time. 

He complained about the errors in the programs and the \ 
examples and felt that the speakers did not outline the capabilities 
of each pijogram - do's and don'ts. Also he was unhappy because 
there was no explanation about the numbers/notation system used 
on the printouts. 

He would like the programs to be available for remote 
terminals and modified so they would take a smaller line (80 
characters vs 132) . He wants programs that are more flexible 
so that they could be used with small computers and suggests that 
programs should take into account small and moderate sized 
buildings. (He designs 5-10 story buildings.) He feels that 
the course should be divided into two secti. is, for beginning 
and students advanced and scheduled for a longer period with 
more hands-on time on the computer. He feels the programs 
require major rewriting to be useful to his company. 

Case#3 - a civil engineer who was familar with the subject 
matter, but not vith the computer programs before taking the 
course. He feel3 he would have gained more from the course and 
understood more if he had some experience and hopes to have 
time this summer to become more familar with TABS. He has taken 
an earthquake analysis course offered by the Structural Engr's. 
Association. 

^ He says he has no need for the programs in his work. His 

firm designs hydroelectric projects and does not do too much, 
structural design. Although he expects to use the TABS program 
eventually, he feels the other programs are too sophisticated. 

He complained that the notes used symbols which were not 
defined in the nbtes; and thinks the notes should include a 
simple example dcine both in the long-hand method and by computer. 
He feels the notes could have been more graphic- 

He would like to take a longer course and commented that 
"Where elso could an engineer learn this without taking a M.S. 
at Cal.?" He feels a semester length course would be good 
because students jneed time to ask questions and work through 
the computer programs on their own. 
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He felt the main benefit of the course to him was that it 
introduced him to uses of four programs and that he learned 
about programs available from other companies. He expressed 
surprise at how inexpensive the programs are to use and liked - 
the opportunity, to get together with other structural engineers, 
to ask questions of the people who wrote the programs, and to 
learn what information NISEE distributes, and the problems 
other us6rs had. 

Summary and Recommendations 

The course was successful in that most of the participants 
were satisfied that it gave them a good introduction to the 
computer codes available for earthquake analysis of shearwall 
buildings, ^s is true in other CEE courses, many students 
who completed this course expressed a desire to take more 
intensive cpbrses on the topics presented - e.g., a longer, 
han(ls-on wo^shop. About half of the students wanted a 
longer progriam, and most of the suggestions for improving the 
course would involve lengthening it or providing alternative 
sections for beginning and advanced students (i.e., more in- 
depth opportunity to work out problems with the computer, 
more opportunity to talk with others who are experienced in 
using the codes, etc.). 

In g.^neral, the course appears to meet the needs of the 
students wno take it, however, the faculty member-in-charge and 
Extension might investigate the feasibility of offering a 
longer course, alternative sections for beginning and advanced 
students, 4nd/or a hands-on workshop as a follow-up to this 
short-course. 
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EARTHQUAKE ANALYSIS OF MULT i STORY FRAME AND SHEARWALL BLDGS. 

Ques t ionna ! re 

The National Science Foundation has funded U.CB. Extension to evaluate the quality 
and usefulness of its short courses for engineers. Your responses to this question- 
naire will help us improve our courses and develop an evaluation model for other 
courses . 

We appreciate your taking the time to answer these questions. 



5. What is your job title? 

Your highest academic degree? dl n,ii Year? 7^-T9 3*o 

6-8. What % of your time is spent in: (i4-7o»/0 
*^-t.o,c [ .1 ] managerial duties boo^tvs^U, 

I %n ] technical duties 

t 1 ] sales/marketing 

9"I2. Do you work primarily in: 

[te^^] Design 6 analysis of structures 
['O] Computer program development 
[ ^i*J Computer applications 
( (-5] Other (speci fy) 

13. How many years has your work involved earthquake engineering? 

-LY g| -y bZJ3 > t Mrs. 

^ XT ^« b 

U*l8. How much experience have you had in using computer analysis techniques 
for designing buildinqs? 

Have written applicable co«Tiputer programs. 
[■JiTi Have performed computer analyses. 

(H ] Have used and interpreted the results of computer analyses. 
[V] Have had other experience. Explcin. 
[/o] Have had no experience. 

19-22. What were your goals in How well were they achieved? 

taking this course? 

V not at all somewhat very well 

2£ [<SJ^ej 
[fcfc] Learning new computer application to 
earthquake engineering 

[U^] updating knowledge of computer appli- 
cations to earthquake engineering [^^] [^^^] 

perspectives for decision-making U [ ^"^^1 [^^^ 

[JW meeting others in the field and 

finding out what they're do^ng [S%] [^o%] 
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Did you find anything in the 
course that you could use 
immediately? If so, what? 




25-26. ETABS ^1 
27-28. DRAIN 20 l4 
29-30. DRAIN-TABS 10 

(On the reverse side, briefly describe W 
valuable and how you used them.) (^,^ i 



Did you find anything in the 
course that you can use in the 
future? If so, what? 

n 

ideas and techniques you found most 



31. How did the pace of the course suit you? (^ij^too fast [73] about right 

] too Slav 

How would you rate the instructors' emphasis on the following? 

too much about right too little 

32. fundamentals V^\\ C^.] <7o 

33. application [X] [4^] j^^j 
3'*. theory [^'o] [7^] (/^j 

35. current research [/o ] [73] [/^] 

36. How did this course compare with other engineering short courses you have 
taken?^ (in usefulness, content, interest) 

[** -4] the best I have taken 

[as] better than others I have taken 

[it] about the same as others 

['S"] less satisfactory than others 

[iS] have taken no others 

37. Who initiated your decision to attend this course? 

(^•7J^you (aV]%your supervisor f^*^ se I ec ted by company 

nr%peiiP ^^r\49k.o One ^ 

Hew does your company support/recognize attendance at courses iike this one? 
l^ffVoiupany strongly encourages attendance 
company pays fees and expenses 
company grants released time (time off) 
{9 \ course attendance Increases possibility of promotion/raises 
[6] no particular encouragement 

Do you plan to seek further instruction or do further individual or company 
research on the topics in this course? 
major company research area 
[r/] plan to do considerable research on my own 
[H'i] plan to take additional courses 
( 71 no 
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'♦0. Do you plan to discuss the information gained in this course with your 
company col leagues? 13^% supervisor? [^'*I'o 

•41. Will the course materials be useful to you in your work? 

expect to use them regularly [S'S^omay use them 
[^% doubt' that jMl use them. 

. k2. Do you think that taking this course will help you professionally? 
If so, how? ^ ^ 6> 

^♦3. How much will your company benefit from sending you to this course? 
[Ill^a great deal h«%somewhat [^%not at all 

Expla i n : 



kk. Would you recommend this course to a colleague? ['S^.no [^Vcyes 
'♦S- What one suggestion do you have for improving tttis course? 

Bib S u^c^e»T*iorSs 

The NSF Evaluation Project Team is planning to conduct case-studies on engineers who 
have taken short-courses. If you are willing to participate in two brief interviews, 
to be scheduled at your convenience, and to permit us to interview your col leagues x 
and supervisor, please fill in your name and phone number below: 

Name 

Phone No.^ 

larea code^ 

{ ] prefer telephone interview 
[ ] want further information 
[ I do not wish to participate 

I do ( ] do not ( ] wish the information I give to be released to my company. 
( ] Please send me a summary of the final report. 
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WRITTEN QUESTIONNAIRE RESPONSES 



Q.l. What is your job ti/tle? 



3 Engineer 

Engineer II 
3' Structural engineer 
2 Project engineer 
5 Civil engineer 

Design engineer 

Designer 

1 Structrual designer 

2 Chief engineer 
Consulting engineer 
Research engineer 

11 Senior structural engineer 
2 Associate engineer 

Associate research engineer 

• 

Managers 

2 Project managers 
Computer * Department 
Chief/ Structural Engineering 

Directors 

Vice-president 
President 

Other 



Staff 




Student 
Professor 
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Q. 2 3-30. Briefly describe tho ideas and techniques you found 
most valuable and how vqu used them. 



1) Response spectrum and fitaVic a.'^alyses are techniques which 
I intend to try t-.o use Ci. future, projects. 

2) Clarification on the de'iails of the software construct — 
very useful in trying to adapt complex models for use in 
analysis. 

3) The most valuable idea is that^the dynamic biehavior of 
the computer models is more dependent on the input ground 
motion than on the type of program or the elements of 

the structural model. Therefore, since the real earthquake 
will not perform per any specifications, I had better 
ignore detailed computer analysis asa source of design 
decisions and design per U.B.C. %/ith all ics "inaccuracies**. 

13) Learning the restrictions of TABS.. ' .■. 

15) I would be very interested in further courses that include, 
othei multi-purpose programs — SAP, EASE, etc. 

24) Course serves as ^ baaic introduction- to the use .of these 
programs. 

28) This two-day seminar reinforces some thoughts which I have 
experienced from time to time, namely that- the structural 
engineer is not paid commensurate to the expertise that he. 
is required to have in order to practice his prof esc ion. • 
He i's too busy chasing the state of the art and wofking, 
and so fa:: hasn't taken the time to beat his own drum and 
carry out a P.R. campaign to let the public know his worth. •« 
\)Z research is necessary to imprpve earthquake engineering, 
but we niay all drive busses if the structural engineer's 
relative worth isn't advertised. (Opinion.) 

30) The main important idea to me is to realize the limitations 
of TABS. It ic dissent ial to know what is not allowed in 
modeling, and what is not a good idea to do. 

37) We use TABS and DRAIN-2D occasionally (usually SAP) . This 

covrse gave some good modeling hints and techniques. Also, I 
gained a greater appreciation for the limitations of these 
programs and I'll be more careful and aware. 

48) TABS & ETABS pr«?sentations were a disaster! Drain 2D was 
excellent by Powell. In any case we feel spending so much 
research money without having the remote terminal use possibility 
is wrong^ especially for average pur shearwall bldgs. where the 
number of D.O.F. is small. 
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Do you think this coyrse will help you professionally? 
If so, how? 

Yes = 26; No or not very much = 3; Possibly =1 

Yes - Any exposure to better technology in this field helps 
you professionally. 

- To avoid errors. 

Enables me to verify design by computer modeling 
should job/client indicate. 

- because (it will enable me tp make) fast decisions 
and fast results in my profession. 

- by applying computer programs. 

- the information was directly applicable to my job. 

- in my favorite field. . . 

- develop sttiictural models. 

- becoming increasingly familar with these programs 
will help me decide which program to use. 

- better unde;rstandiiig of capability of programs. 

- as knowledge of structural behavior of buildings 
increases more desim^ emphasis will be needed to 
use this knowledge J This class helped me gain 
knowledge I can use. 

- It will enable me to solve design problems in a 
' better manner. 

- There have been many developments since I ^graduated 
in/ 72. 

- To use available computer programs for building design. 

- but not in the immediate futute. 

- to further my background in the state of the art 
research and application of structural engr. programs. 

. - 2 have not had previous exposure to Berkeley programs 
which are generally accepted by the engineering field. 

- current analysis procedures help design and productivity. 
It gives a better view of analysis techniques that 

y are available. 

, - I gained insights that will be useful in technical 
analysis and review of dynamic analyses and in 
government contracts. 

- it increases my range — what I can do. 

- it may improve the efficiency of my depart;ment through 
the use of these progreuns. 

gives me more tools, 
-by keeping up with advancements. 

- broadened my knowledge in codes other than Sap4 and 
gave 'me a better concept of non-linear behavior. 

- I know the capabilities and limitations of TABS & ETABS. 

- The knowledge of current research and use of computer 
■programs is important to. structural engineers. 

- We use computers all the time and increased familiarity 
with these types of programs can .only help. 

- Now have a bade understanding of dynamic application. 

Not unless I pursue better explanations by better course 
presentations. 

Not very much - need instructors who are better prepared & 
generally better at their jobs. 

No - not much for now - there is no direct applicability 
it this time. 

Possibly - get a job and/or start my own outfit. 

: : iQo 



Q.43. Continued 



- The supervisors all are very familar with 
these types of programs and the assumptions 
involved. It will only help me. 

- older members of the office feel less com-, 
fortable with computer programs. They would 
like someone to be able to understand then. 

Not at all - not enough practical explanations and 
applications presented to benefit liQh* 
company. 

- The goal of my company in sending me to 
this course is to develop a skill using 
minicomputers to solve structural design 
and not using existing programs. 
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How will your company benefit from sending you to this course? 



Ans.: A great deal = 7; Somewhat = 16; Only a little = 1; 
Not at all = 2. 

May improve the efficiency of my department through use of 
these programs. 

A great deal - I will pass this information on to all 

others in our firm. 

- We're in the process of developing computer 
analysis^ 

- No one at our company is that familar 
with TABS. 

- it will improve the capability in earthquake 
design. 

- the use of these programs is increasing. 

- we are on the threshold of designing major 
concrete ductile frame building (40 stories high 
and up to 60) and need to understand and 
per^rm sophisticated analysis to increase 
judgment and capability. 

Yes - I became .iware of alternatives and realize that we 
do not have in-house capabilities to use programs. 

Only a little. 

Somewhat - knowledge of current research analysis techniques 
are important to our program. 

- the supervisors are all very familar with these 
types of programs. The course will only help me. 

- many of our current projects are too small for 
these applications or they're made of wood. 

- the cost of course is so high that it will be 
a long time before my benefits from using 
what I learned outweigh cost. 

- there is no direct applicability at the present 
time. 

- we use, or will use# these programs occasionally. 

- our company product is not directly related 
to computer progrcuns. 

- I'll hold in-house seminars. 

- we don't design many high rise buildings in 
our office. i 

- I suspect the time required for learning input/ 
output data will be reduced. ^ 

- only a small % of our work involves structural design 

- We would usually pay Oshraf to do this I 

- it will increase our capability. 

- I will be able to be involved in inelastic • 
analysis projects. 

- We use TABS & Drain 2D occasionally (usually SAi') . 
This course gave some good modeling hi :s and 
techniques. Also I gained a greater appreciation 
of the limitations of these programs & I'll be 
m(^e careful and aware. 

19-^ 
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Q.45. what one suggestion do you have for improving this course? 

1) Spread i£ out over many evenings with access to computer to 
\ experiment and bec9me familiar with software discussed. 

3) Make it less expensive. (69 participants x $225 « $15,525) 

4) Send notes beforehand fpr familiarity. 

5) The speakers describing the input and output must be more . 
professional than academic — to keep interest in listening 
and understanding. 

7) There should be long-term courses emphasizing applicability 
of computer pi:ograms. 

8) Add workshop for actual usage of programs. 

9) Seems OK--next would be workshop. 

10) Write brief explanations cn^coThpu^er sheet. 

11) Provide more background information relating to previous 
developments by MT^vl^ading to present state, of art for 
betTOr continuity. 

12) *A bit more background in fundamentals. 

13) Find a better location. 

15) More practical application of modeling techniques. 

19) Better preparation of reference material. 

20) More emphasis on useful tools. 

21) 3 days instead of 2. 

24) Examples should be of a smaller size so we can take 
examples into our office to test run. 

25) More emphasis on topics not covered in manuals — advanced 
problems, program characteristics, (stiffness degradation) — 
distribute manuals prior to course so basic information would 
not have to be covered in such detail. 

26) Lower cost to make available to more people. 

27) I had problems jrelating a practical use for "Drain" in its 
current state of development. 

28) Put audience 'in the ballpark when subject matter is changed. 
Several speakers were so involved in their topics that chey 
were difficult to follow. When speakers jump around, they 
tend to assume audience has same familiarity with topic or 
example problem that thiey do. A little more qualifying 
would be useful. 

29) More examples of modeling and modeling techniques. 

30) For me, the info, on DRAIN was good for background, but 
too much, time was spent on it compared to its practicality. 

31) More streamlining, better preparation of lecturers. 

32) The sketches to the distributed computer output should have 
been copied. 

33) . Nonel 

35) Course should emphasize more the benefits to practicing 
engineers. 

37) Stress modeling techniques, but not actual "how to input." 
Anyone can read the manual for input. 

41) More than 2 days. 

42) More and slower presentation on non-linear behavior and 
program assumptions. 

4 3) Get practical engineers to present material. 
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Q.45. (Continued) 

45) One day longer (3 days). 

46) present information on the .pros and cc^s of\various programs 
available to the practicing eiigineer, e.g., cost comparison 
of several commercial programs, presented somewhat like a" 
consumer guideline. 

Present useful information of damping values to various 
types of structures that a practicing engineer may encounter. 
Techniques on smoothing a response spectra that is attained 
from a soil-s engineer, for input into a program. In other 
words, basic practical techniques required by the analyst ^ 
to perform a response spectrum or time history analysis. 

47) Instructors who are better prepc^red and generally better 
at their jobs. ■ . 

48) Emphasize basic assumptions. Do's and don\ts. Also how to 
use TABS, ETABS for shearwall bldgs. (no frame). Also, gear 
TABS for terminal users. 

• • • 

Suggestions From Interviews 

1. Found there were problems with sign convention^ in math 
models and a few things that weren't included" in. the UCp programs 
(e.g., can't include cantilever structure in program) . 

* t 

A better way to present information is' to have a work session 
for individual to run sample problem himself. 

The DRAIN (inelastic) prog raips^ are not yc*t useable because 
the physical processes are not we fKunder stood; ' A client would not* 
setf the results as being reliable. ^ . ' ' 

• • 

2,. A semester- long CEE course on similar topic. Need good fixfst- 
hand knowledge of topic to get a lot out of a short course. Need 
a longer course. Someone commented that "Where could an engineer 
learn this without taking a MS at Cal?" A semester CE course would 
be good - need time to ask questions and work through computer 
programs on your own. 

3. 94% of the instructors and people are from California and 
already aware of programs (he's from Colo.) and the class should 
last longer because we need to get on the computer. Programs 
presented are powerful for large buildings butjheglect small and 
moderate-sized buildings. Programs should be <tiore flexible so 
they can be used with small computers. (T. design 5-10 story bldgs.) 
The course needs to be divided into two seminars - one for beginners 
and one for advanced people. 

4. Might spread course out over evenings for several weeks so 
everyone could try out programs in between. 
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5. Possibly publish a booklet a»ay not be possible because field 
changes rapidly. 

6. Information presented did not always fellow notes*- did not 
tie together w|l] . • \ 

7. Physical set-up was very good, but should have some time for 
engineers to discuss between 1st and 2nd day. People who had to 
catch planes on the seoopd day d^d not have time. to ask questions. 

8. Biggest handicap was getting up in' a.m. We sJibuld have 
better directions to get to class. 

9. Need mcrre convenient location and lower fee (I'm' a private* 
consultant) . • 

10. Limitation in bomputer programs, could be pulled out of 
context and delineated better. 

11. The inelastic development was useful only as knowledge and 
not very practical now (these take too much computer time, and are 
useful only in research, not design). I could have skipped that 
day. , • . * . . 

12. Still a little too much like a textbook, but that was due 
to lack of time. ♦ 



er|c 



202 



EVALUATION OF JHE ENGINEEkING DESIGN IN TIMBER COURSE 
BY THE UNIVERSITY i3F CALIFORNIA ENGINEERING EXTENSION DEPARTMENT 

July 18-2^1, 1979 



Evaluators: Martha Maxwell, Ph.D. 

NSF Project Director 

Rick Boettcher, 
Research Assistant 



Engineering Design in Timber is a 3)i-day course intended for engineers and 
architects who possess a general knowledge of structural theory and design 
problems but who need the specific technology of designing in timber. The course 
covers strength of timber in relation co grading and use, and the particular way 
j.n which, timber properties affect structural problems, including use of sawn 
timber, plywood, and g-lued laminated Construction. Lectures include the grading 
and production of timber and glulam; beams, columns, shear walls and diaphragms; 
timber connectors and structural systems. 

# 

Professor Jerome M. llaphael, University of California Structural Engineer- 
ing and Structural Mechanics, was the faculty coordinator, assisted by four 
instructors Thomas E. Brassal of the Institute of Timber Construction, Stephen 
A. Mahin of U.C. Berkeley, Ron Sanchez of the 'American Plywood Association, and 
Kenneth D. Smetts,.a consulting structural engineer. 
. • 

This evaluation is based on the responses of 28 students who filled in a 
questionnaire on the la^t day of the course and on telephone interviews with 
nine students called abouc a month after the course ended. 

' Charlie teri sties of the Students . A varied group of people attended this 
£ourse*-engineers,. professors, architects, draftsmen^ sales personnel and others. 
Nineteen were college graduates and 8 had earned a master's degree. The average 
student completed his last college degree in 1968 and has worked in timber design 
for over 5 years. However, there was a 30-year range with 40% of the group hav- 
ing ^less than one year's experi-ence in timber. Most enrolled in the course on 
their own initiative and with company support. 

Student Evaluations , Jtost students took the course to improve their general 
background and increase their skills and felt the course met these goals well. 
Those who were interested it) Icfaming fundamentals rated the course as succeeding 
fairly well, but those interested in acquiring a perspective for decision making 
or meeting others in the field rated it only somewhat satisfactory. Overall those 
who. had taken other short courses rated this one as slightly better than other 
courses they had taken. Tue average student felt that he was well prepared aca- 
demically for the course in thpory and fundamentals, but less well prepared in 
practical experience. 

This project was funded by the National Science Foundation under 
Grant *SED- 78-221 3^. The opinions, finding, conclusions, and recommenda- 
tions expressed are those of the authors, and do not necessarily reflect 
those of the National Science Foundation. 
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Most students found sojpcthing of practical use to theb in the course. 
Plywood technology was se^as most useful iiancdiately and the glulan infor- 
mation as more useful in |he future than other topics. Most rated the in- 
structors' emphases on reiearch, theory, and fundamentals as about right but 
felt there was not enougn emphasis on applications. Most say they will use 
their notes and the course handouts regularly. 

Forty- one percent of the class plan to take other courses on the topics 
of t\\is course and 29% plan to do considerable research on the^ir ovm. Most 
felt the course helped them professionallyand 67% said they wo^ld recommend 
the course to others. Those who would not recommend it said ths^ it was not 
applied enough. 

Student suggestions for improvii g the course included passing put out- 
line notes of the lectures ahead of time, including more information on prob- 
lems and solutions with real world applications (for example, designing a 
house showing all connections) and extending the course for another day. Others 
suggested that the speakers go into more depth on some topics--which would re- 
quire a longer course. One suggested that the speakers be videotaped and that 
the overhead aids be xeroxed for the students. 

Information from the interviews illustrate the widely different needs of 
the students: 

Student A is a senior structural i^ineer who supervises 8 people 
and works for the state of California/ -He felt his preparation in theory 
and structuie was above average, and took the course to improve his 
technical information, and to learn more about costs. He feels the course 
helped him quite a bit. Prior to this he took courses given by the 
American Plywood Association and one on pre-stressed concrete given by BART. 
He Tints this course the best he has taken. He has shared his notes and 
handouts with his 8-pers&n staff and brainstorm 5d with them about ideas 
from the course. He tock notes and has used them. He feels he's better 
informed about the state of the art as a result of talking to the con- 
sultants and listening to the presentations. He enjoyed the course very 
much and says he will take it again in two or three years. He suggests 
having a dinner and speaker. 

Student B has been a professor of engineering for 32 years but has 
never taught a timber course. He went into this course cold except for 
theory and basics. He will teach a course in design in a year and wanted 
to prepare himself, he has taken a related course at UCLA Extension and 
was impressed that the UCLA course was better organized and i^at th^^ 
professor distributed course notes. He felt Raphael should have had more 
sample problems, but thnt he was just not prepared- -'lie's well qualified, 
but not adequately prepared with materials." He said this course had a 
good balance between industry and university people. His major complaint 
is the lack of notes. He doesn't like to take notes madly and would like 
to have them given out to students. He rated the course as about the 
same as other courses he has taken although he said he did learn how to 
begin to design timber struptures. 

J 
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Student C has no experience in timber design and said he was 
starting from zero. He works in structural engineering, but does 
not work much with timber. He took the course to prepare for the 
state Structural Engineer Exam and thinks he did pretty well on the 
test. If it hadn't been for the class, he says he would have been 
in dire straits to pass the exam. The only related course that he 
has taken was on the design of offshore structures. He would like 
to have xeroxes of the overhead*^.^ He felt the instructors were well 
prepared but varied in their ability to answer students questions, 
and thought that Smetts assumed a lot of knowledge on the part of 
students. He was very well satisfied with the course, and rated it 
better than other short-courses he has taken. 




Summary and Recommendations: 



The students enrolled in this class differed widely in background, experi- 
ence, interests and their reasons .or taking the course. Most were satisfied 
that the course helped them improve their understanding and updated their knowl- 
edge of timber design, as ^ell as giving them some ideas/ techniques that they 
could use on their jobs. The main complaints were that some of the speakers 
did not emphasize applications enough particularly the academic instructors, who 
did not present enough practical examples, problems and solutions to illustrate 
the points in their lectures. Al:.j there were complaints about the need for 
course notes and additional materials. 

Since this was the first time this course was offered, there is room for 
improvement. The following are some suggestions: 

4 

1. Speakers should prepare examples, illustrations, and practical 
problems to illustrate their concepts so that the practicing engineer 
students can relate to and understand the information they present. A 
significant number of the students in this class have no experience in 
timber .design and cannot make the transfer from theory to practice with- 
out explicit examples. 

2. Course materials should include outlines of the presentations 
and copies of the viewgraphs used* Engineers who take short-courses 
expect and need outlines of the academic/ technical presentations. With- 
out outlines, the tiewcomers to the field have problems following the 
new information and become frustrated in their attempts to learn. Out- 
lines are especially necessary for presentations on difficult, abstract, 
or highly technical information. 

3. The course seems Co mee^^ an in)portant need for people in timber 
design, and the faculty coordinator should consider ways of changing the 
course so that it better meets the range of student needs--perhaps by 
extending it for a half-day or planning some optional sessions where 
students can choose between basic fundamentals and more in-depth presen- 
tations. 
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JOB TITLES 

Number 

5 Civil Engitteers 

3 Senior Engineers 

7 Structural Engineers 

2 Professor Civil Engineers 

1 Assistant Engineer 

2 Architects 

1 Bridge Engineer 

1 ^ Design Draftsman 

2 Sales Representatives 

1 Water Resources Engineer 

1 Operations Mgr* 

1 President, Facility Supervisor 
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27. Comments: Was there something you expected to learn but didn't? 

more extensive lateral analysis (i.e. loading) 
residential construction 
shear wall connections 

structural problems such as: column splices, checked beam shear 
when proper to use, procedure 

methodology 

practical design prohiems 

economic aspect of designing with alternate methods 

more connection details, and want more examples, than theory 

detailing and design for lateral forces 

timber piles, timber underground and underwater 

load design and distribution 

1) Practical consideration of detailing connections 

2) Pre-engineered truss 

3) Good practice in connection design to allow for shrinkage 

52/53. Comments: Will this course help you? Would you recommend it to 

others? 

recommend for Arci Seniors as update 

help keep up with changes in code/manufactured prod. 

prepared to work w/professionals 

knowledge for decision-making 

new data in handouts useful 

help in passing structural engineering exam for Cal . 

20 V 
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Q, 52/53. (cont'd) 



have taught a similar course several times and I wanted to 
learn how others taught it. Get new information. 

increase proficiency in design 

access to resource persons for assistance 

better 'understanding of timber design. 

Q- 54. Participants' comments: Suggestions for improvement. 



Strongly urge that all lectures be previously prepared and notes 
passed out. Very important. Impossilbe to keep up with professor. 

Typed notes ahead of time, outline. (3) 

Course geared to building construction. More emphasis should be 
put on fact that timber is an important material in heavy con-> 
struction. Its use is subject to other sp^cs such as AASCHO, 
AREA, etc. There are some differences in applying mechanics. 

Increase length. (1) 

Practical applications (3) 

Instructors had to rush. Some lecturers were too fast 

Found most valuable the information on glulam design, plywood 
properties and design. 

Methodological approach, example, problems needed ^ 

* 

Should include a spund discussion on problems and solutions 
Tuition high 

Give it 2 weekends c • 

More practical. Use designers as teachers; less theory-oriented 
professors 

Too broad brush: concentrate on specific areas 

Real world applications needed to make courses more comprehensive 
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Q. 54 (contM) 



Eliminate Mahin; have Raphael and Smetts do the course almost 
by themselves with one lecture only from tn industry 
representative Smetts is the most important because he is the 
only do^signer. 

Shorter lectures 

Spend a day designing a house showing all connections 
Smetts and Sanchez very good 
Mahfin good 

Biassel average; Raphael poor! 

More emphasis on derivation of loads and correction factors. 
Extend to 5 days 

Add 1/2 day to pro-engineered truss and economy of frame 



choices 
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ENGINEERING DESIGN IN TIMBER 
J\m% 1979 Class 
Questionnaire 
Re sponde n t s« 2 8 



The National Science Foundation has funded U.C.B. Extension to evaluate 
the quality eaxd usefulness of its short courses for engineers. Your 
responses to this questionnaire will help us improve our courses and 
develop an evaluation model for other courses. 

We appreciate your taking the tlM to ans%#er these questions. 

5. What is your job title? Structural engr. /civil engr.«65%;architect« 
draftsinan«lS%; sales"10%; other (professional^etc. )«10% 

6-8. Your highest academic degree? Y rr -^t j . \ted 
Bii-19; MS-8; 1-HS > 

r- 7^ / 178 

0. Have ycu had any previous courses i' - r 
Yes -9 (32%) 

lO-I?. What % of your time !?pe * 

Avera^, (28%) managerial duties 

(29%) design 

(21%) stx'uctural engineer . ' 

(:3%) other? (specify) 

13-14. For how many year/ haff your oi ^ . ^ ' ^ri.^ . • Lgn? 
average* 5.3 yrs, Ran9e 1 tD 30 yrs • . ,v; ; ^'0% less than 1. 

15-20. What were your goals in LoJcing thi." ' ' \a order. 

Inmost important, through 5«least important.; 

% rating each go^i;:! as luust Important. Note zcum rated more than one as 
most important 

(21%) learning fundamentals of a field new to me 

(54%) improving general background 

(50%) updating technical Icnowledge 

(4%) acquiring a perspective for decision-ma)cing 

(4%) meeting others in the field and finding out what they're 
doing 

( ) other (explain) 
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21.-26. How well did the course satisfy your goals? (Check) 
* Not at all Somewhat Fairly well Very well 

^7 Learning fundamentals 



26 Improving background 

27 Updating tech. knowledge 

26 Acquiring perspective for 
decision making 

22 Meeting others in field 

Other (what?) 



7% 
7% 



(33%) 
(15%) 
(15%) 
(35%) 

(64%) 
( ) 



(41%) 
(58%) 
(45%) 
(35%) 

(32%) 

( ) 



(26%) 
(27%) 
(33%) 
(23%) 

(4%) 

( ) 



27. Was there something you expected to learn in this course, but 
didn't? If so, what? Yes 46% (See list) 

28-30. HOW well prepared were you for this course in the following: 

^ average (l"poorly; 3«well) 

theory 



fundamentals 



practical experience 



2.4 
2.5 
1.8 



Did you find anything in the course 
that you can use immeidiately? If so, 
what? 



Did you find anything in the ^ 
course that you can use in the 
future? If so, what? 



31-32. Wood Species & Grading 

33-34. Sawn Timber Beams 

35-36. Glued Laminated Beams 

37-38. Glulam Systems 

39-40. Plywood Technology 

41-42. Columns 



17 
17 
14 
13 
21 
14 



5 
9 
11 
5 
4 



On the reverse side, briefly describe the ideas and techniques you found 
ttiost valueible and how yoti used them. 

(See list) 
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How would you rate the instructors* enpasis on the following 



43. 


current research 


(4%) 


(85%) 


too little 
(11%) 


44. 


fundamentals 




(78%) 


(3%) 


45. 


application 


(4%) 


(44%) 


(52%) 


46. 


theory 


(4%) 


(81%) 


(15%) 



47. How did this course cocppare with other engineering short course you 
have taken? (in usefulness, content, interest) 



(11%) the best I have taken 
(28%) better than others I have taken 
(36%) eUsout the sane as others 
(21%) have taken no others 

48. Who initiated your decision to attend this course? 

(75%) you (11%) your supervisor (14%) selected by covc^Ay 

49. HOW does your company support/recognize attendance at courses like 
this one? 

(25%) company strongly encourages attendance 

(71%) coiq>any pays fees and expenses 

(36%) cooqpany grants released tine (time off) 

(4%) course attendance increases possibiliy of promotion/raises 
(14%) no particular encouragement, paid my own way 

50. Do you plan to seek futher Instruction or do further individual or 
company research on the topics in this course? 

(14%) major coiic>any research area 

(29%) 'Plan to do considerable research on my own 

(41%) plan to take additional courses 

(18%) no 

51. Will the course materials be useful to you in your- work? 

(68%) expect to use them regularly 

(29%) may use them 

(3%) doubt th^t I'll use them 

52. Do you think that taking thi? course will help you professionally? 
If so, how? Yes-25 No-3 

53. HOW many others in your company/ branch work in timber design? Average 3.6 
Would you recomment this course to them? Ye8-67% No- 33% (because it lacked 
practical information.) 
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54. What suggestions do you 



Other remarks? 



Questionnaire, Page 4 

» 

have for improving this course? 
(See attached) 



The NSF Evaluation Project Team is planning to conduct case-studies on 
engineers who have taken short-courses. If you are willing to participace 
in t%#o brief interviews, to be scheduled at your convenience, and to permit 
us to interview your colleagues and supervisor, please fill in your name 
and phone number below; 

Nam e 

Phone No. — 

(area code) 



( ) prefer telephone interview 
( ) want further information 
( ) do not wish to participate 
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Evaluation Report on the U. C. Extension 
Computer Aids for IC Technology and Device 
Design Program Held July 9, 1979 



Kirtha Maxwell, Ph.D., Evaluator 
Rick Boetcher, Research Asst. 



Questionnaires were mailed on August 29 to a random sample of 100 par- 
ticipants who enrolled In the July 9, 1979 U. C. Extension Progrem on Computer 
Aids for IC Technology and Device Design. This report summarizes thz responses, 
of the 52 people who replied to the questionnaire. The number of people re- 
sponding Is small, and may not represent the views of the total group of 300 who 
attended the program, therefore' the results siiould be Interpreted with caution. 



Characteristics of the Participants 

The respondents, represent a highly educated group about equally divided 
between those holding bachelor's, master's and Ph.D. degrees. They completed 
formal education over a long time-span — between 1949 and 1979 with the average 
graduating In 1972. The average respondent has worked In his present specialty 
years and spends about 20% of his time In administrative duties, SOX In re- 
search, and 20% In production or other. Only one Is a full-time administrator 
while nine are full -time researchers. The average respondent supervises one or 
two other employees. 

Seventy- two percelit of the group said they were familiar with SUPREM be- 
fore taking the course, 60% had used It, and SOX plan to use It In the future. 

Sixty-five percent said their attendance was self-Initiated and 35X were 
selected by their companies to attend the program. Program fees and expenses 
were paid for by their compfinles In 98X of the cases. Twenty percent reported 
that they were the only representative of their company Who came to the program, 
while 16S stated that their company sent more than 10 representatives. 

Seven people had taken the course before and 4 of these said there was 
enough new Information to keep them Interested. 

Thirty-five percent felt they were well-prepared for the class while 27X 
felt unprepared. Sixty- three percent said that It would have helped them If 
course materials and notes had been sent to them In advance of the program. On^ 
said that he felt very well prepared because he had /ead the materials and notes 
from last year's program and discussed It with colleagues who had attended It. 
Some said that having course materials In advance would also Improve the question 
Ing period. 



This project was funded by the National Science Foundation under 
Grant # SED-78-22138. The opinions, findings, conclusions, and rec- 
onnendatlons expressed are those of the authors, not NSF's. 
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Thirty-four percent found all of the topics presented would be useful 
in their work, others found several topics useful. Only one respondent re- 
plied that nothing from the course could be used in his job. 

Seventy-two percent said they were familiar with SUPREM, 60S had used 
• t. and 80% plan to use it in the future. 



Participant Evaluation 

The average respondent was very well satisfied that the program achieved 
their goal of getting an overview of current research. On the average, re- 
spondents were less satisfied that the program met their other goals — i.e., 
learning fundamentals, learning current theory, and solving process-problems. 
These they rated as only "somewhat" attained. Those who expected to learn to 
solve process problems were the least satisfied. Also those who were interested 
in meeting or contacting others in the field were disappointed as there was 
little time for this (e.g., a video-tape was shown during the lunch hour. 

Althougli 54* of the group thought the program description was accurately 
stated, six of these said the program title was incorrect because it implied 
that the instructors would spend more time and put more emphasis on process 
modeling and the computer aids than they did. Fifty-eight percent of the re- 
spondents said they expected to learn something other than the program offered. 
(This represents a much higher percentage of unmet participant expectations 
than we have found in other CEE short-courses.) 

One outcome of CEE short courses is that they stimulate participants' 
interest in learning more about the topics presented. This course was no ex- 
ception. Participants gave about 40 topics on which they would like to see 
Extension plan courses. Perhaps the large number of responses. to this question 
reflects the frustration of those who attended primarily for process modeling 
information and found minimum emphasis on this. 

Participants made a number of suggestions for improving the program. 
Complaints centered around two areas — the limited time devoted to process 
modeling and computer programs and the course materials. Thirty percent of 
the group, including a large number of Ph.D. participants, complained about 
the lack of detailed outlines of the lectures — saying that it was impossible 
to take adequate notes of the fast-paced, technical presentation. Perhaps they 
have a greater need for the details of the experimental information in their 
work or perhaps they learn best from reading. Others complained that the ma- 
terials given out in the program were not related to or did not follow the con- 
tent of the lecture. Although 29?; of the respondents felt the course materials 
would be very useful in their work, 27% considered them of little or no value. 

Some participants were deeply disappointed, others angered at the long 
delay in receiving lecture notes promised by the program coordinator. 
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Participants In this course were highly educated, experienced engineers 
with a great deal of Interest In the course topics and a desire to learn more. 
Those who expected to get an update on Stanford's latest research In IC Tech- 
nology were well satisfied; but those who were Interested primarily In device 
design processes and In learning more about the computer aids were unhappy with 
the limited time spent on these topics. Other than the ccroplalnts about the 
course materials, most suggestions were directed at providing more Information 
on some or all of the topics addressed In the program — suggesting that the 
program could be lengthened. 



1. Course Title . Either the title should be changed to something like A One- 
Day SymposTum on Current Research In IC-Technology . or If the current title 
is used, more instructional time and emphasis should be given to process- 
modeling and the computer programs. 

2. Improve Course Notes and Materials . Detailed, hard-copy outlines of the 
lectures should be given to each participant at the start of the program. 
Copies of all viewgraphs should be Included In these materials. If these 
and other course materials were sent to participants In advance of the 
program, many would benefit and perhaps the complaints on course content 
and emphasis would be reduced (provided, of course, that the title Is 
accurate.) 



Engineers who enroll In Intensive, highly technical short programs like this 
one expect and need to receive outlines of the lectures, viewgraphs. and 
other materials. Without these aids.. memory fades fast and the probability 
of the participants' recalling the Information when needed Is greatly re- 
duced. Currently, most successful programs offer these. 

3. Schedule More Opportunity for Participant Involvement . Since the majority of 
respondents had used SUPrEM and many would welcome an opportunity to discuss 
their experiences and question the Instructors about current developments, 
changes, future plans, etc.. more discussion time shouU be planned. This 
might be done by extending the program an extra day o«%With1n the one-day 
•ormat by scheduling discussion periods or question-answer sessions at the 
same time as some of the research presentations and letting the participants 
choose which session they wished to attend. As one respondent observed. "An 
open session involving user Interchange and experiential feedback would be 
most helpful; possibly more so than a yearly update of some of Stanford's 
other research work." 



RECOMMENDATIONS 



Since the present course materials are very useful to some part1c1p< 
materials supplementary to tne lectures and presentations should be 
continued. 
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Also "breaks" of adequate length and lunch peridds should t£ scheduled 
free of formal presentations to all participants to talk to others and 
relax a bit. , 

UC Extension should plan additional courses that fol low these brief .state- 
of-the-art programs. Perhaps extra sessions the following day in the form 
of workshops on the computer programs, or a series of short foTlow-up 
programs that address some of the course topics in greater depth. 
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COMPUTER AIJS FOR IC TECHNOLOGY AND DEVICE DESIGN 
Course Questionnaire - July 9i 1979 

.The National Science Foundation has funded U.C.B. Extension to 
evaluate the quality and usefulness of its engineering short-courses. 
Your responses to this questionnaire will help us improve our programs. 

We appreciate your taking the time to answer these questions. 

1. Who initiated your decision to attend this course? 

(t^fl^I did ^)^niy superviser (— ) I "ivas selected by 

my company 

2. How doep^^our company encourage attendance at course like this? , 

pays fees and expenses- J 
gives time off / 



^ pa 
(si) gi 



(— ) pays part of expenses ^ 
(:Js^ special support 

3- Hew many people from your company are attending this course? 

^. What is your highest academic degree? 6*«K«l«ri' . 31% . (M«»Tw'r37% 
When did you receive it? (year) * 

5. What is your job 'title? 

6. What ^^o^^our time is spent in» 

(iSlT Supervision/ Administration 
(sb^ Research / 
(/d'it) Production Processes 
j(- ) Other (What?) 

7. Hew many people do you supervise? o^Hi*^.; dx.fs'l; s-r-- 

8. What were your goalgii^n taking How well were they' achieved? 
. this course? (Rank usl'.g #1 as 

most important) Not at all/ Somewhat . Very well 

2. ( ) Learning fundamentals ('1 )% (P*.)% (5 )% 

it , ( ) Overview of current (A> ) (<f/ ) (^'7) 

. research 

^ i ( ) Learning advanced theory (it) (ty) (a<j) 

* ( ) Solving process problems (aj) ^uf) ( 3 ) 

Other (specify) mc«.t.n-| ort«4** -i i ) " (^ ) (- ) 

9. How well prepared were vou frr this course? . .. ^-fc, , i*iO> 

10. How much would it havfit^neiped you if course materials were sent out 
in advance? Mo^ * k * '* »» %; %^me\Jk,Jt , xi**^ t <io*A w^^Vcr>, mueh^aslL 

11. Have you taken this course (or one like it). 'before? 

'When? 

If yes, was there^ enough new matjsrial to k«ep you 
interested? oui <r^-7 ^c«|>on»«,. -'f ^<,,a "^-'j 3 *' n/' 

/ 
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12. What specific topics are most related to your work? (CVD, Implantation 
3h'>« jsh.a "em". t etc . ) 



10. 



How long have you been working in this area? cujtAoqe r '< u-^ 
Was the course description stated accurately? 



14. Was there something you expected to learn, but didnt? 
If so, what? ti»-*»re*>o 

^ - »fi.% 

15. Did you have specific problems you wanted help on? " , 

16. What information will be of most use to you on the job? 

17. Were you already familiar with SUPREM? 

Have you used SUPREME? — to*). 

Do you plan to use it in the future? - g.^^^ ,^.)^ ^ «j 

18. How useful will the course notes be in your work? 

19. . nat topics would you like to have covered in other Extension 
Course's? s««. . 

/ 

20. Where else do you have a chance to learn more about your field?- 

company conferences (^7) in-house (*» ) inter-company 

('o) Berkeley (<J) Stanford (9 ) other universities 

(7) extension (-) colloquiums {/f) grad/undergrad . 

courses 

21. What suggestions do you have for improving this course? 

« 
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other remarks! 



Return tot 



Engr. Ext. 
U.C. Berkeley 
2223 Fulton 
Berkeley, CA 9^720 



The NS? Evaluation Team is planning to arrange follow-up interviews 
with participants of this course to assess long-term usefulness and 
job applicability of the information covered in this course. If you 
are willing to. take part in a brief interview, scheduled at your 
convenience, please fill in your name and phone number below. 



Name 



Phone HQtng \{ , ) 



Work I ( ) 



( ) Want further information. 
( ) Do not want to participate 



( ) prefer phone interview 
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Responses to ^. 12. *Vhat specific topics are most related to 
your work? 

all (12) 
^ CVD Impl. (15) 

Laser anneal. (6) 

Imp. difusion tind oxidation kinetics (5) 
iS^vice analysis & process modelti (6) 
3UPRSM & SEDAN (2) ' >^ 

Compu. modeling (2) 
ZFI & Poly sii etching (1) 



Responses to Ik. Ms there something you expected to learn, but 
didn't? 

No- 16; yes-21 

Wore on SUPREM - data on experimental verifiation ; history 
and status of each user function; update on SUPREK & 
future plans; what' was wrong with present models, etc. (8) 

More on process modeling (IC) 

Less about device design (1) 

Solving device charac -eristics rather than process parameters (1 
Detailed device modeling on short-channel MC3 (1) 
Exact avaialbility of SEDAN, cost, etc 



Responses to Question 15. Did you have specific problems you wanted 
help on? 

yes- 17; no-28 ; n.r.-7 
.Vhat is punch through preceise:'y? 

Solving device characteristics from process parameters. 
2-d bipolar modeling, 
laser annealing, 
arseniz model in SUPREi*! 
MCS Modeling 

Details on SUFREM use (3) 
y.ore accurate model in Vt 



^2 0 
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Replies to Q. <Vhat topics would you like to have covered in other 
Extension Courses? 

photolithography (OPT, E-Beam) 

failure analysis techniques. 

advances in thin SIO2 film formation methods 

high electromigratlon metalurgical systems 

numerical analysis & modeling of semiconductor devices 

more detail on device analysis and computer slfaulation. 

modeling 

more-detail on laser annealing 

circuit k logic simulationtechniques. Device modeling for simulaticn*. 
device physics fundamentals} cv analysis as an analytical tool, 
materials characterization} process induced defects, 
derivation and use of SUPREM and new work on device design models • 
data base for process modelling' 

high speed device (e*g*t short channel MOS technology 
photovoltaic device technolojgy 
etching Si and other materials 

each of the specific topics of this seminar could have a course 

devoted to it. 
a^anced oxidation theories. 

problems and solutions in high surrent ion implementation 
problems and solutions in plasma etching. 
CAD / j 

acre of the same^. ' 
experimental verification of SUPREM 

Uses of SUPREM and examples. Parts of the program explored in detail. 
SEDAN, TANDEM . 
SUPREM arsenic n^odel. simulating implants through nitride and oxide/ 

nitride layeirs>epi autodoping model* 
two-dimension afid one-dimensional field and current distribution 

modeling. Of major interest to me would be a course on the 

electrical chaiacteristics of Si02 in particular, conduction 

and charge tirapping' 
2-d modeling for computerized circuit analysis. 
MOS modeling 

getter ing, high pressure, oxidation, enhanced diffusion. 

PN junction reverse-bias failure modes 1 mobile charges, surface 

staten, precipitate emission gettering, stacking faults, 
advanced lithography) plasma processing 
the same. 

More detail on the TANDEM program. 
Scaled MOS 

More users information on SUPREM 
short channel effects 
power MOS overview 
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""'""^h^^ It^r in^or-ation .iu be of „oBt us, to you on 



the 

All the information was helpful (2) 
General background info, on IC Tech. 
Update on Stanford Research 



progress at 3taf . ICT 



Recent 

Update on Stanf . & Hitachin dev. 
Cuhlbert (?) research (1) 
iiUPREM (rel. research) (5) 



Lab. (1) 
sim. 



prog. (1) 



Laser annealing & oxide info. (5) 
Physical mechanisms of prcess steps in 
Physics & math of device modelling (1) 
2-dim. modelling as described in course 
Lack. of progress in 3-d modelling 
Cxid. .theory & ion implant- 4 
How best to interpret modelling results (1) 

or ^^Either correlative data 
Good course notes would be (2) 
Techniques used by other programs. (1) 
Don't know (1) 
none (2) 



SUPREM (1) 
lit. (1) 
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Responses to Question 21 » 

What sugf;estions do you have foi- improving this course? 

Replies by Ph.D. Participants i 

1. Course Materials! 

— The material handed out and the lecture were completely 
different. I have yet to use the material, frinted notes 
would have been a greater help to those attending and added 
more to the question periods. 

--Provide notes of talks'. The information handed out at the 
conference was largely old. Prof. Dutton did say that 
course notes would be mailed out. I have not received them. 

--The course materials handed out will be of very little use 
to me since they didn't reflect the content of the lectures. 

— Hand out course notes of the lectures with good details. 

— Course materials should be more closely related to the 
actual lectures. 

-- Make the course materials relate more closely to the talks. 

-- Have the course materials be available to the participants 
in aetail and in advajnce of the course. 

-- I am deeply disappointed in the fact that the course organizer 
does not even bother to send a copy of lecture notes to the 
attendees. I am extremely disturbed because the course notes 
distributed in the class did not reflect the exa/st content 
of the lectures. 

-- Handing out actual lecture notes at the start of the course 
is a must. This was not done, and much information could 
not be written down fast enough. The Handout was not well 
related to the lecture material. 

-- Copies of viewgraphs are a must i preferably as nandouts 
before course begins. Lack of the latter was a huge disap- 
nointment. ^ r 

Responses from participants with Master's/Bachelor's Degrees. 
—Send viewgraph copier,. 
— Better notes. 

— Handout notes before course. 

— Copies of the lecturers' notes would be useful during the 
presentations for better notetaking. Some of the material was 
presented too quickly for note taking. I have not yet received 
a copy of the speaker's notes which would be most useful. 

2. Course Content and Emphasis » 

I would prefer more emphasis on CAD^mcra^ls. This was a very 
minor part of the program. I had expected more emphasis 
on CAD models from reading the cqurse brochures, but less 
than 20';o of the class was spent on this. 

— From the course brochure, I expected something on detailed 
device modeling of short channel MOS - this was not covered. 

— As a C£ii£8£ it could be better balanced. The content emphasis 
currently reflects areas of recent Stanford activity. 1 
expected more information on device modelling/simulation 

from the description in the brochure. 
-- Not enough details on modeling. 
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ReeporiiioJ to 4- 21 (Cont'd) 

Uo into the actual use of SUPREft* such as limitations, 
tradeoffs, run efficiency, etc. Of course, this could 
be a course in itself. 

— Jhould have a good introduction for JUPREM beginner. 

— Cut down on amount of material presented & increase Basic 
Material Science Content* 

I think most engineers are interested in availability 
of models than can be used jifiw (fully realizing that 
they aren't perfect.) 
--1 expected more about SUPREM - the history and status 
of each user function. 

— The subject drifted from computer aids. I expected 

more information about the availability of SEDAN - cost, inc. 
— Follow outline more closely (i.e., content of course < 
should follow initial courseVescription? Expected more 
about the use of computer model— how to use SUPREM. 

— I liked the course. However, many people with no process 
background attended this course because of the misleading 
title. They were not very satisfied. I expected actual 
program description. 

— I expected more details on the status of modeling research 
at Stanford and wanted more details on use of SUPREM. 

—This was a report by graduate students on SUPREM - not at 

all like the flyer described. Expected more on MOS modeling. 
— Break up course into several parts in order to put greater 

emphasis on participants* area of interest. 
--I thought it was going to be on computer . modeling. It was 

not so much a course as a conference. Course description 

stated inaccurately. 

— Never use videotape. 

— Expected more descriptive information on device modeling 
activites. 

An open session involving user interchange & experiential 
feedback would be most helpful t possibly more so than a 
yearly update of some of Stanford's other research work. 

— Overall the course was very worthwhile lor me and I am 
planning to attend next year. 

— Course over-emphasized process —device line. Expected 
more on process relatir^j to device characteristics. 
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Responses to Q. 21 (Cont'd) 

Course content & emphasis 

--This was not a course but a forum for S.U. staff to 
tell us what they were doing and show us i|ow clever 
they were regardless of course title, needs of attendees, 
and outrageous fees. I expected to learn about the deriva- 
tion and use of SUPREK and new work on device design models. 

I expected to learn how exactly SUPREM & TANDEM work. We 
need a hands-on demonstration at the computer terminal. 
— Have two courses - one general and one advanced for those 
working in the field of modeling. Make faculty accessible 
for questions. 

— The course title was misleading. Course was mostly about 
IC Technology and very little about computer modeling of it. 

— I would have liked more detailed information on laser anneal- 
ing. • . 

—How about a course in circuit and logic simulation techniques. 
Device models for simulation? This' course was very well 
organized and presented. 

— I expected to hear less about device design. I'm interest- 
ed in learning more about the data base for process 
modeling. 

-- I expected more on SUPREI/j. 

-- I was looking forward to more information on the use 
of computers in IC Tech & D vice Design and wish more 
emphasis had been placed on'^that. Overall it was 
fairly good and broad. 

— I wanted more time to be spent on device modeling and 
much less time on KCJ Processing as ^-as described in 
the course description. 

--Too much for one day; should be a two-day course. 

Instructors 

— I have no suggestions. Both content and presentations 
were excellent. 

--Eliminate talks like the one B.^:. Deal gave. 

— Laser annealing vould have been more useful had the 

speaker. Gibbons, attended. 
--I expected to learn the latest problems with and update 

on oU^'REK. 
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EVALIATION OP Tf£ U. C. BEIUCElEy QdSlSION CXXJRSS 
"INIERFACING TD A MICROOOMPOTIR- 
(JUne 18-24, 1979) 



Instructors: 

Professors David Patterson, Carlo S4quin 
Oepartnent of Electricad Bngineering 
and Ocnputer Sciences, University of 
California, Berkeley 



Evaluators: 
Martha Maxwell, Ph.D. 
NSF Project, Director 
Rick Boettcher, 
Research Assistant 



Interfacing to a MicrocGRpitet is a five-day hands-on lab course de- 
signed to give participants an overview of the multitude of input/output 
interfacing techniques and of the hardware/softwere tradeoffs involved; anl 
the experience necessary to design the microoorputer interface to the real 
vrorld for their own particular applicatixxi. IVo instructors lectured each 
tnoming and the participants spent the afternoons working in the lab. on 
six experiments with the help of lab. assistants. Ihe foUowing wete listed 
as prerequisites for the course: 1) engineering or scientific background; 
2) seme ptoyiauiiing experience (in any language), and 3) seme fandliarity 
with TIL integrated circuits. 

TSiis report is baaed on the responses of twelve participants to a 
questionnaire given on the last day of class, interviews a month after the 
course with 4 participants, observations of the class, and discussions witli 
the instructors. 



Characteristics of the Students 

All students were college graduates. One a a^naster's degree. They had 
been out of college an average of 8 years. On the average, they spent 41% 
of their time on research and developnent activities, 6 percent on test/ 
quality assurance and 20% on production. Four persons were engaged in other 
activities including technical writing, software design, administration, and 
design. About half of the class work in equipnent fabrication, 41% design 
programs and 33% test p r o gr a n a while one-quarter perform all three functions. 
Fifty percent of the class had no previous experience in interfacing periph- 
eral devices with ooRfuter systems and 5 considered themselves well prepared 
or fairly well prepared for the oourde. In general the class was somewhat 
diverse in experience and background ranging from amateurs to those actively 
working with oanputetd on relevant projects. 

Evaluations of the -Course 

Most of the class were quite well satisfied that the course had offered 
them a better understanding of micro-devices, a perspective for decislon- 
maicing, and an opp or t u nity to expand job opportunities. Those who were in- 
terested in exploring new career edtematives were scmewhat less satisfied. 

This project was funded by the National Science Foundation under 
Grant #SED-78-22138. The opinions, finding, conclusions, and 
reoonmendations expressed are those of the authors, and do not neces- 
sarily reflect those of the National Science Foundation. 
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"Interfacing to a Microaaqputer" , page 2 



A third of the class expected to learn scroething different than the course 
offered - i.e., nore details of hardware techniques, nore on applications, 
more ocnparative data on miccoooRiJuters vs available oonfuters. 

In general, students v«ere very well satisfied with the course. Itiose 
«to had taken previous oour -a rated it as better ttvui other courses they had 
taken. 

The students all rated the instructors as good to excellent, found 
the lab sessicsns very helpful, and the lab. assistants excellent. Most 
students found SGoe topic in the course that they oould use imnBdiately and 
some (about 33%) felt that the topics would be useful in the future. Also 
they were pleased %d.th the course materials. 

Student ccanplaints center e d around the pcobleros they had getting the 
aof there to work, machine "glitches", etc. Some felt they needed more of a 
cookbook approach and/or more explanations of conputer acronyms. 

The fbllcwing interview sumnaries illustrate seme differences between 
students, their range of interats, and the applications they plan to make: 

Student A is a ccnputer scientists %d.th a BS in engineering who works as an 
analyst/pKo yraiu ii ei in biomedical research. He took the course because he 
is interested in the actual interfacing of hardware. !llthough he h^l little 
back^praund, (none in electrical engineering) he rcw feils pretty oonpetent 
about micro-processing a jd '^ell able to find reference materials. He is tak- 
ing other g3»^Jate courses towerd a degree in coopiter science. He rated the 
course as th$ best he has taken and the instructOLii as excelleni: ani plans 
to take other courses in extensinn. He is using tho course neiterials regular- 
ly and feels that the course har been particularly valuable to him profession- 
ally since he was limited in knowledge about hardwere before taking the course. 
He thinks the CEE courses are son?«<tfKat ^lo^etical (or a: plied at a high level) 
but that this is good because they are state-of-the-art aixl offer a quick way 
of bringing people up to par. 

Student B is a systems anaJ/st with a strong background in programning, but 
little JowwledgTi of hardware. He took the course because he was Interested 
in learning more about hardware and to help him in his project cf developing 
a general purpose modular data access system. He has not used the information 
learned in tha course, but expects to use it and plans to take other courses 
(provided the^' don't require a heavy EE background.) 

He felt the course had too mudi information on microprocessor architecture 
(z-8000) and felt he needed more information on interfacing. He conslderdd 
the lab. too short and felt much time wes wasted on machine "glitches". He ■ 
stated chat he learned quite a bit and rated the course as better t^ian others 
he had taJcen. He plans to take additional courses and has had 80i« chance to 
spread the information to S or 6 others in his department. 
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"Interfacing to a Micrtxxanjutef", page 3 



Student C is a technical writer with a recent Bachelor of Arts degr<»<5 who 
took the course to be able to understand operations, tQ expand her present 
capabilities and to explore the possibility of going into pcogramidng. She 
felt she knew a fair anount about mic rop rocessors before taking jJie course, 
but had no actual exp^ience nor training in oonputers. After taking the 
course she felt very well prepared to start learning progrannlng aid has 
since taken a .pcogranmlng course. She feels well prepared for it because 
she had learned the basics in this miczoprooessing course. 

She feels that taking this course has made her writing more inforroa- 
tive and that she is much better able to convey Infxartant ideas. She does 
not feel the need to take more courses in Interfacing since she doesn't need 
to invent or create material — only to understand it. She rated thfe In- 
structors as good and the lab. assistants great, but said she needed more of 
a cookbook set of instructions for tte harduere part of the course. 

Student D is a design engineer who said he knew a bit about miczoprooessing, 
but not a lot before taking the course. He took the course to determine 
vhetiier the technology would be useful to his OGnpany. He has not used the 
material because his con^iany is not in a position to use micxopcocessing as 
yet. He feels his knowledge about the subject doubled as a result of taking 
the course, and that the Ir.structors and lab. assistants were excellent. He 
rated the course as very good and feels the information he leam^l %dll be 
beneficial to him In the future. 

Sannary and Reu mme ndatlons 

Although this class varied In their expertise and sophistication with 
oonputers, the students all gained something they wanted from the course and 
considered the course successful. Ohey were pleased with all aspects of the 
course ~ the lectures, labs.. Instructors, and assistants, the aall size 
of the class and the Intensive personalized attention given In the labs, made 
it possible for individuals to learn what they needed. 

Ihe reccmnendatlons suggested are to find ways to resolve the diffi- 
culties with the soft-were (or to help students accept these difficulties 
as inevitable); and perhaps provide those who are unfamiliar with the computnr 
terminology a glossary of terms and 
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ItmiTPCIM TO A MIiCROOQM>UTER 
Course Questionnaire 
J\]ne 2^,1979 

N - 12 



The National Science Foundation has fisvted U.C. BerJceley Ebctension to evaluate 
the quality and usefulness of its short courses for engineers. Your responses 
to this questionnaire will help us iBfxave our courses and develop an evalua- 
tion model for other courses. 

Wb apireciate your talcing the tine to answer these questions. 

1-4. What is your highest academic degree? year graduatid? 

BA/&S -11 MS * 1 average 8 yrs. out 

of sdiooJp 

5- 12. Wiat percentage of your work involves: 



(41) R&D 

( 6) Test/Quality Assurance 
(20) Production 
(23) Other mat?) 



13. Do you; 



(42) design piujiams 

(50) fabricate equipnent 

(33) test p i.-ogi a B g 
all (25) 

14. Are you primarily a; 

I 

i 

(2) supervisor 

(0) rdsearch associate 

(5) engineer 

(2) technician 

(3) other (what?) 

15. Vat many people do yom supervise? 

X - 3.5 

16. How much opportunity' is thwre for to dissoninate the information 
you learned in this course to others in your company? 



2.8 



(1-4 range) 



17. Hew lOLch egqpurience did you have in interfacing peripheral devices with 
CGR^Hiter systems before taking this course? 

6 (or 50%) none 2 or very little 2 fairly 1 more than average 

18. How well prepared were ^ for this course? 

not sufficiently - 1, seme amateur 3, fairly well 2 (had taken other 
1 in hardware 2 very well 1 don't low courses) 
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19. In your opinion v«re the course prerequisites stated aocurately? 

yes "'8 2 * no (probably should have had 8080 lang. & experience 
2 « don't know or no opinion 

20-8 . What v«re your goals in taJcing How well were they achieved? 

this course? (Rank order using (Check fron very well tc> not 

#1 as the most importeuit, 2 as at all,) 

X Not at aU Very Wbll ^ 

(8) acquiring a perspective for . , ' (0) (0) (3) (3) (4) 4.1 
decision-making 

(8) expanding job capabilities 3.7 (1) (0) (2) (5) (2) 

(ICj) getting a better understanding of 4.2 (0) (0) (1) {7i O) ^ 
j micro-devices 

(6p exploring a new career alternative 3.3 (1) (0) (2) (2) (1) 

(p Other (what?) interface to real world (1) 

Was'there something you expected to learn in this course, but didn't? 
If so, what? 4 yes 7 no 

^0-38. Mhat topics in the course will be of most use to you on the jdb? Either 
iflinediately or in the long run? 

of iaropirliate use useful in the future 

Program Operations 3 4 

Interfacing with Lnput 

& Output Devices 9 1 

Data handling capabilities 3 4 

Mi c ro pioc essor operations 5 4 

(8080 chips) 

(On the reverse side of this pagf - please describe the ideas 6 techniques you found 
most helpful and how you will use them.) 

See comnents 

39. How did ti'ls course conpare with other short courses you have taken? 

% 

(20) the best I have taken 

(50) better than others I have taken 

( -) about th^ same as others 

( -) less satisfactory than others 

0^) have taken no others 
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Interfacing to a Nicxoocmputer, 



Oomnents ranged fran "gcxad" to "excellent" 
(see list attached) 



40. Please ocnment on the following aspects of the course: 

Instroctnrs 
Lectures 
Lab. sessions 
Lab. esqjeriinents 
Lab. assistants 

41. Hew useful will the text and course materials be in your work? 

see oomnents ^« 

42. What suggestions do you have for inpaxving the course? 

see oomnents 

43. Are there other courses that you'd be interested in taking? 

see oomnents 



The ^6F Evaluation Project Team is planning to conduct case-studies on engineers 
vto have taken short-courses. If you are willing to participate in two brief 
interviews, to be scheduled at your convenience, and to permit us to interview 
your colleagues and supervisor, please fill in your name and phone ninber below. 



Name 



Phone # 



(area code) 



(6) prefer telephone interview 
(6) do not vant to participate 



(1) went fiffther infozmation 
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glDDEWTS WRITKilj a]WlP^JN QUESTIONS ' 

Q. 29. Was there something you expected to learn in this cxxarse, but didn't? 

If so, what? ' ^ 

Yes - 4 

(2) ybre coitparison information to help choose a specific 
np like 6502 vs 6800 vs 8080. PDP, HP 300, etc. 

(1) More on applications on processes are suitable for. • 
(1) Details of hardware techniques. 

No = 7 

Q. 30-38 Which techniqtris and ideas did you find most helpful^tifij\aw will you 
use than? 

-a general understanding of conputer architecture and the 
capabilities of np. 

-details of machine language and assg. language programing- the 
power of the instruction set. 

-details of how performance will increase in the next few years. 
-Vo techniques necessary to interface thru Qprts/a keyboard, 
CRT, and TTY with a 6502 np. 

r 

Q. 40 OoniK^ts on aspects of course: 
Instructors: 

The hardvkcure instructor made his material yery clear, hswever 
I found it difficult to leap around as much as the instructor 
in software although the material is more familiar to me than 
hardworc. 

In&.b:uctors and lectures weie good, but labs, weren't co^^ed 
enough in cla^. 

Instructors were excellent and lectures very good. 

Instructors were knoK^ledgeable and answered questions thoroughly 
fX3 that the student could understand. Th^r were good. 

Lectures were well organized and understandable. 

Instructors were very good to excellent - clear and logiccd. 
Ihe lectures were goqd, of necessity skiinning & fast. 

Instructors were excellent and very good. Lectures were good, 
but fast. 

Instructors were excellent; material well organized and presented 
to coincide with experiments. Lectures could have included a 
little more detail re 8080 instructions. 

Instructors and lectures were both good. 
Instructors and lectures were both very good. 
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Students Written Corments on Questions 

Q. 40. Lab. Sessions & C>q3erinents: 
-quite good (1) 

-needed nore time to do out-of* -class vnrVperfoiin experiments. (2) 
-Lab. assts enthusiastic 
-all very good / helpful (8) 
-excellent (1) 

-need better introduction to the H.P. terminal 8080, 

material should be oondensod, i.e. exactly does a certain O.S. 
subroutine do? Lab. assistants were very good. 

-Ebqieriments were well designed, but hcd too many problems with 
ths equipnent. 

-E>q)eriments were well chosen and of progressive difficulty. 

-Could not have completed labs, without the lab. asst's help. 

-Lab. sessi'^'^s acceptable, as were the lab. experiments. 

-Lab. sessions were a good learning experience. Experiments were 
challenging, but good. Some frustration with the reliability 
of equipnient. 

-Needed more guidance, perhaps in the form of an actual demonstration 
prior to each experiment might have been useful. They were en- 
joyable experiments, however more cookbook instructions (partic- 
ularly for hardware) would be useful. 

-Lab assts. were great! Very helpful, knowledgeable & encouraging. 

-Labs had too little' guidance relative to vAiat peripheral equipment 
was doing (What was in the "black box"?). 

Q. 41. Hew useful will the text and course materials be in your work? 

y ^ probably useful 
^ very useful - it gives me a solid background for understanding 

good 
unknown 

very useful. The text can well be used as a reference aid 

as it is well organized and indexed. 
Will use it as a 1st reference. 
Text will be helpful . - 
Seme 

very useful for reference. 

Q. 42. What suggestions do you have for inp:x)ving the course? 

' -Discussion of some specific areas v^ere mp are being used -nd the 

advantages gedned. 

* • 

* -Better lab. equipnent, list of specific "tricks" needed to get 

software to voric - more consistency between actual hardware 

and docunentation. 
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Q. 42. (continued) 



-none 

-More guidance in lab. , perhaps in the form of an actual 
dotonstration prior to each experiment might have been 
useful. They were enjoyable experiments, but more cook- 
book instructions (particularly for hardware) would be 
useful. 

In describing periphery chips, etc., explain sane of the 
acronyms used in ocirputer science. 

Q. 43. Are there other courses that jou'd be interested in taking? 

progranming microprocessors 
yes- unspecified 

oonpilersAculd like to receive notices of future courses, 
dont know of any others, 
not at present. 
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